Accurate and Efficient Statistical Testing for Word Semantic Breadth

• URL: http://arxiv.org/abs/2605.08048v1
• Date: Fri, 08 May 2026 17:38:36 GMT
• Title: Accurate and Efficient Statistical Testing for Word Semantic Breadth
• Abstract summary: dispersion-based statistics serve as proxies for contextual diversity.<n>We propose a Householder-aligned permutation test to isolate differences from directional differences.<n>Our method reduces Type-I error by 32.5% while preserving sensitivity to genuine breadth differences.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measuring the breadth of a word's meaning, or its spread across contexts, has become feasible with contextualized token embeddings. A word type can be represented as a cloud of token vectors, with dispersion-based statistics serving as proxies for contextual diversity (Nagata and Tanaka-Ishii, ACL2025). These measurements are useful for deciding appropriate sense distinctions when constructing thesauri and domain-specific dictionaries. However, when comparing the breadth of two word types, naive hypothesis testing on dispersion can be misleading: differences in semantic direction can masquerade as dispersion differences, inflating Type-I error and yielding "statistically significant" outcomes even when there is no true breadth difference. This is problematic because significance testing should distinguish genuine effects from incidental fluctuations in small-difference regimes. We propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. Our method applies a single Householder reflection to align the mean directions of the two word types and then performs a permutation test on the aligned token clouds, yielding calibrated, non-parametric p-values. For practicality, we introduce a GPU-oriented implementation that batches permutations and linear algebra operations. Empirically, our alignment reduced Type-I error by 32.5% while preserving sensitivity to genuine breadth differences, and achieved a 23x speedup over the CPU baseline.

Related papers

• Is Inference Mediated by Distinct Semantic Structures in LLMs? A Mechanistic Interpretation [34.50304918655722]
  Transformer representations are known to carry label-level information, but whether they encode semantic operations producing those labels is unclear.<n>We investigate this in Natural Language Inference using controlled premise-hypothesis pairs that differ by a single semantic transformation.
  arXiv  Detail & Related papers  (2026-05-25T07:21:07Z)
• Amortized Variational Inference for Partial-Label Learning: A Probabilistic Approach to Label Disambiguation [2.7214777196418645]
  Partial-label learning trains classifiers when each instance is associated with a set of candidate labels, only one of which is correct.<n>We introduce a novel framework that directly approximates the posterior distribution over true labels using amortized variational inference.<n>Our method employs neural networks to predict variational parameters from input data, enabling efficient inference.
  arXiv  Detail & Related papers  (2025-10-24T09:54:23Z)
• Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression [102.24287051757469]
  We study self-supervised covariance estimation in deep heteroscedastic regression.<n>We derive an upper bound on the 2-Wasserstein distance between normal distributions.<n>Experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.
  arXiv  Detail & Related papers  (2025-02-14T22:37:11Z)
• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis.<n>Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• SoftCVI: Contrastive variational inference with self-generated soft labels [2.5398014196797614]
  Variational inference and Markov chain Monte Carlo methods are the predominant tools for this task.<n>We introduce Soft Contrastive Variational Inference (SoftCVI), which allows a family of variational objectives to be derived through a contrastive estimation framework.<n>We find that SoftCVI can be used to form objectives which are stable to train and mass-covering, frequently outperforming inference with other variational approaches.
  arXiv  Detail & Related papers  (2024-07-22T14:54:12Z)
• Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings [17.803726860514193]
  Detection of semantic variation of words is an important task for various NLP applications. We argue that mean representations alone cannot accurately capture such semantic variations. We propose a method that uses the entire cohort of the contextualised embeddings of the target word.
  arXiv  Detail & Related papers  (2023-05-15T13:58:21Z)
• Learning Counterfactually Invariant Predictors [11.682403472580162]
  We propose a model-agnostic framework, called Counterfactually Invariant Prediction (CIP) Our experimental results demonstrate the effectiveness of CIP in enforcing counterfactual invariance across various simulated and real-world datasets.
  arXiv  Detail & Related papers  (2022-07-20T09:23:35Z)
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests [87.60900567941428]
  A spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character. We study stress testing using the tools of causal inference.
  arXiv  Detail & Related papers  (2021-05-31T14:39:38Z)
• Statistically significant detection of semantic shifts using contextual word embeddings [7.439525715543974]
  We propose an approach to estimate semantic shifts by combining contextual word embeddings with permutation-based statistical tests. We demonstrate the performance of this approach in simulation, achieving consistently high precision by suppressing false positives. We additionally analyzed real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus.
  arXiv  Detail & Related papers  (2021-04-08T13:58:54Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• Wasserstein Distance Regularized Sequence Representation for Text Matching in Asymmetrical Domains [51.91456788949489]
  We propose a novel match method tailored for text matching in asymmetrical domains, called WD-Match. In WD-Match, a Wasserstein distance-based regularizer is defined to regularize the features vectors projected from different domains. The training process of WD-Match amounts to a game that minimizes the matching loss regularized by the Wasserstein distance.
  arXiv  Detail & Related papers  (2020-10-15T12:52:09Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.