A Fisher's exact test justification of the TF-IDF term-weighting scheme

• URL: http://arxiv.org/abs/2507.15742v2
• Date: Thu, 24 Jul 2025 19:47:16 GMT
• Title: A Fisher's exact test justification of the TF-IDF term-weighting scheme
• Abstract summary: Term frequency-inverse document frequency, or TF-IDF, is arguably the most celebrated mathematical expression in the history of information retrieval.<n>This paper justifies the use of TF-IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Term frequency-inverse document frequency, or TF-IDF for short, is arguably the most celebrated mathematical expression in the history of information retrieval. Conceived as a simple heuristic quantifying the extent to which a given term's occurrences are concentrated in any one given document out of many, TF-IDF and its many variants are routinely used as term-weighting schemes in diverse text analysis applications. There is a growing body of scholarship dedicated to placing TF-IDF on a sound theoretical foundation. Building on that tradition, this paper justifies the use of TF-IDF to the statistics community by demonstrating how the famed expression can be understood from a significance testing perspective. We show that the common TF-IDF variant TF-ICF is, under mild regularity conditions, closely related to the negative logarithm of the $p$-value from a one-tailed version of Fisher's exact test of statistical significance. As a corollary, we establish a connection between TF-IDF and the said negative log-transformed $p$-value under certain idealized assumptions. We further demonstrate, as a limiting case, that this same quantity converges to TF-IDF in the limit of an infinitely large document collection. The Fisher's exact test justification of TF-IDF equips the working statistician with a ready explanation of the term-weighting scheme's long-established effectiveness.

Related papers

• TF-IDF and BM25 Are Exact KL Divergences [0.0]
  We show that both scoring methods admit an exact interpretation as Kullback-Leibler divergences between two probability models.<n>The resulting framework provides a common theoretical basis for TF-IDF and BM25, clarifies what they measure, and allows them to be compared theoretically with other information retrieval methods.
  arXiv  Detail & Related papers  (2026-09-12T16:02:02Z)
• Diversity in Large Language Models under Supervised Fine-Tuning [15.119045051735633]
  Supervised Fine-Tuning (SFT) is essential for aligning Large Language Models (LLMs) with user intent, yet it is believed to suppress generative diversity.<n>In this study, we attribute the decline to two primary drivers: the forgetting of preexisting knowledge and low-frequency patterns within finetuning datasets.<n>Our evaluation confirms at scale that generation narrows after SFT and strengthens the hypothesis explaining this effect.
  arXiv  Detail & Related papers  (2026-04-30T20:20:59Z)
• Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness [1.9349283460893496]
  We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness.<n>We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks.
  arXiv  Detail & Related papers  (2026-04-01T09:17:01Z)
• All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection [67.89888669159899]
  RFC Bench is a benchmark for evaluating large language models on financial misinformation under realistic news.<n>The benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis.
  arXiv  Detail & Related papers  (2026-01-07T18:18:28Z)
• Association via Entropy Reduction [0.0]
  We provide a different score, aver, and observe, on a dataset with ground truth marking for association, that aver does better at finding assciated pairs than tf-idf.<n>We observe that (1) aver has a natural threshold for declaring pairs as unassociated while tf-idf does not, (2) aver can distinguish between pairs of documents for which tf-idf gives a score of 1.0, (3) aver can be applied to larger collections of documents than pairs while tf-idf cannot, and (4) that aver is derived from entropy under a simple statistical model.
  arXiv  Detail & Related papers  (2025-11-07T01:03:48Z)
• A Unified Analysis for Finite Weight Averaging [50.75116992029417]
  Averaging iterations of Gradient Descent (SGD) have achieved empirical success in training deep learning models, such as Weight Averaging (SWA), Exponential Moving Average (EMA), and LAtest Weight Averaging (LAWA) In this paper, we generalize LAWA as Finite Weight Averaging (FWA) and explain their advantages compared to SGD from the perspective of optimization and generalization.
  arXiv  Detail & Related papers  (2024-11-20T10:08:22Z)
• Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods [59.779795063072655]
  Chain-of-Thought (CoT) prompting and its variants have gained popularity as effective methods for solving multi-step reasoning problems. We analyze CoT prompting from a statistical estimation perspective, providing a comprehensive characterization of its sample complexity.
  arXiv  Detail & Related papers  (2024-08-25T04:07:18Z)
• Flow matching achieves almost minimax optimal convergence [50.38891696297888]
  Flow matching (FM) has gained significant attention as a simulation-free generative model. This paper discusses the convergence properties of FM for large sample size under the $p$-Wasserstein distance. We establish that FM can achieve an almost minimax optimal convergence rate for $1 leq p leq 2$, presenting the first theoretical evidence that FM can reach convergence rates comparable to those of diffusion models.
  arXiv  Detail & Related papers  (2024-05-31T14:54:51Z)
• Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning [73.75282761503581]
  We propose DiffTPT, which leverages pre-trained diffusion models to generate diverse and informative new data. Our experiments on test datasets with distribution shifts and unseen categories demonstrate that DiffTPT improves the zero-shot accuracy by an average of 5.13%.
  arXiv  Detail & Related papers  (2023-08-11T09:36:31Z)
• A Comparative Study on TF-IDF feature Weighting Method and its Analysis using Unstructured Dataset [0.5156484100374058]
  Term Frequency-Inverse Document Frequency (TF-IDF) and Natural Language Processing (NLP) are the most highly used information retrieval methods in text classification. We have investigated and analyzed the feature weighting method for text classification on unstructured data. The proposed model considered two features N-Grams and TF-IDF on IMDB movie reviews and Amazon Alexa reviews dataset for sentiment analysis.
  arXiv  Detail & Related papers  (2023-08-08T04:27:34Z)
• Method for Determining the Similarity of Text Documents for the Kazakh language, Taking Into Account Synonyms: Extension to TF-IDF [0.0]
  The task of determining the similarity of text documents has received considerable attention in many areas such as Information Retrieval. The term frequency - inverse document frequency (TF-IDF) is the most widely used term weighting method to facilitate the search for relevant documents. The effectiveness of the method is confirmed by experiments on functions such as Cosine, Dice and Jaccard to measure the similarity of text documents for the Kazakh language.
  arXiv  Detail & Related papers  (2022-11-22T15:54:41Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Lower bounds in multiple testing: A framework based on derandomized proxies [107.69746750639584]
  This paper introduces an analysis strategy based on derandomization, illustrated by applications to various concrete models. We provide numerical simulations of some of these lower bounds, and show a close relation to the actual performance of the Benjamini-Hochberg (BH) algorithm.
  arXiv  Detail & Related papers  (2020-05-07T19:59:51Z)
• TF-IDFC-RF: A Novel Supervised Term Weighting Scheme [0.0]
  Sentiment Analysis can be applied in several contexts to classify the attitude expressed in text samples. The most popular term weighting scheme is TF-IDF (Term Frequency - Inverse Document Frequency) This work presents a comparative study on different term weighting schemes and proposes a novel supervised term weighting scheme, named as TF-IDFC-RF.
  arXiv  Detail & Related papers  (2020-03-12T21:31:46Z)
• Semantic Sensitive TF-IDF to Determine Word Relevance in Documents [0.0]
  We propose STF-IDF, a novel semantic method based on TF-IDF, for scoring word importance of informal documents in a corpus. Our method managed to decrease the TF-IDF mean error rate by a factor of 50% and reaching the mean error of 13.7%, as opposed to 27.2% of the original TF-IDF.
  arXiv  Detail & Related papers  (2020-01-06T00:23:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.