Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness

• URL: http://arxiv.org/abs/2604.00672v2
• Date: Sun, 05 Apr 2026 09:41:22 GMT
• Title: Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness
• Abstract summary: We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness.<n>We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks.
• Score: 1.9349283460893496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: TF-IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks. This paper provides insights into TF-IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.

Related papers

• A Fisher's exact test justification of the TF-IDF term-weighting scheme [0.0]
  Term frequency-inverse document frequency, or TF-IDF, is arguably the most celebrated mathematical expression in the history of information retrieval.<n>This paper justifies the use of TF-IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective.
  arXiv  Detail & Related papers  (2025-07-21T15:54:23Z)
• The Consistency Hypothesis in Uncertainty Quantification for Large Language Models [22.60039074743706]
  Black-box uncertainty quantification (UQ) methods, relying solely on model API access, have gained popularity due to their practical benefits.<n>In this paper, we examine the implicit assumption behind several UQ methods, which use generation consistency as a proxy for confidence.<n>We propose data-free black-box UQ methods that aggregate similarities between generations for confidence estimation.
  arXiv  Detail & Related papers  (2025-06-27T01:53:15Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• A unified Bayesian framework for interval hypothesis testing in clinical trials [4.911220423050305]
  The American Statistical Association (ASA) cautioned statisticians against making scientific decisions solely on the basis of traditional P-values. We demonstrate that the interval null hypothesis framework, when used in tandem with Bayes factor-based tests, is instrumental in circumnavigating the key issues of P-values.
  arXiv  Detail & Related papers  (2024-02-21T16:01:06Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Exact Distribution-Free Hypothesis Tests for the Regression Function of Binary Classification via Conditional Kernel Mean Embeddings [0.0]
  Two hypothesis tests are proposed for the regression function of binary classification based on conditional kernel mean embeddings. Tests are introduced in a flexible manner allowing us to control the exact probability of type I error.
  arXiv  Detail & Related papers  (2021-03-08T22:31:23Z)
• MASKER: Masked Keyword Regularization for Reliable Text Classification [73.90326322794803]
  We propose a fine-tuning method, coined masked keyword regularization (MASKER), that facilitates context-based prediction. MASKER regularizes the model to reconstruct the keywords from the rest of the words and make low-confidence predictions without enough context. We demonstrate that MASKER improves OOD detection and cross-domain generalization without degrading classification accuracy.
  arXiv  Detail & Related papers  (2020-12-17T04:54:16Z)
• On the Replicability of Combining Word Embeddings and Retrieval Models [71.18271398274513]
  We replicate recent experiments attempting to demonstrate an attractive hypothesis about the use of the Fisher kernel framework. Specifically, the hypothesis was that the use of a mixture model of von Mises-Fisher (VMF) distributions would be beneficial because of the focus on cosine distances of both VMF and the vector space model.
  arXiv  Detail & Related papers  (2020-01-13T19:01:07Z)
• Semantic Sensitive TF-IDF to Determine Word Relevance in Documents [0.0]
  We propose STF-IDF, a novel semantic method based on TF-IDF, for scoring word importance of informal documents in a corpus. Our method managed to decrease the TF-IDF mean error rate by a factor of 50% and reaching the mean error of 13.7%, as opposed to 27.2% of the original TF-IDF.
  arXiv  Detail & Related papers  (2020-01-06T00:23:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.