PromptNCE: Pointwise Mutual Information Predictions Using Only LLMs and Contrastive Estimation Prompts

• URL: http://arxiv.org/abs/2605.21776v1
• Date: Wed, 20 May 2026 22:10:54 GMT
• Title: PromptNCE: Pointwise Mutual Information Predictions Using Only LLMs and Contrastive Estimation Prompts
• Abstract summary: Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings.<n>We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities.<n>We introduce a benchmark with human-derived ground-truth PMI across three publicly available datasets, and evaluate five information-theoretic prompting-based estimators.
• Score: 3.015315464409211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings. We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities. We introduce a benchmark with human-derived ground-truth PMI across three publicly available datasets, and evaluate five information-theoretic prompting-based estimators. Our main method, PromptNCE, frames conditional probability estimation as a contrastive task and augments the candidate set with an explicit OTHER category. We show theoretically that adding OTHER recovers the true conditional P(y | x) rather than just a ranking over listed candidates, turning a contrastive prompt into a general-purpose zero-shot probability estimator. PromptNCE is the best zero-shot method on all three datasets, reaching Spearman correlation up to 0.82 with human-derived PMI. We also present a case study in computer science education showing how these estimators can be used to score student knowledge summaries in a low-data setting.

Related papers

• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Leveraging Large Language Models for Predictive Analysis of Human Misery [1.2458057399345226]
  This study investigates the use of Large Language Models (LLMs) for predicting human-perceived misery scores.<n>We evaluate multiple prompting strategies, including zero-shot, fixed-context few-shot, and retrieval-based prompting.<n>To move beyond static evaluation, we introduce the "Misery Game Show", a novel gamified framework inspired by a television format.
  arXiv  Detail & Related papers  (2025-08-18T07:02:59Z)
• Probabilistic Reasoning with LLMs for k-anonymity Estimation [32.33315973296769]
  We introduce a new numerical reasoning task under uncertainty for large language models.<n>We propose BRANCH, a new methodology that estimates the k-privacy value of a text-the size of the population matching the given information.<n>Our experiments show that this method successfully estimates the k-value 73% of the time, a 13% increase compared to o3-mini with chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2025-03-12T17:41:25Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Semi-supervised Learning For Robust Speech Evaluation [30.593420641501968]
  Speech evaluation measures a learners oral proficiency using automatic models. This paper proposes to address such challenges by exploiting semi-supervised pre-training and objective regularization. An anchor model is trained using pseudo labels to predict the correctness of pronunciation.
  arXiv  Detail & Related papers  (2024-09-23T02:11:24Z)
• Query Performance Prediction using Relevance Judgments Generated by Large Language Models [53.97064615557883]
  We propose a new Query performance prediction (QPP) framework using automatically generated relevance judgments (QPP-GenRE)<n>QPP-GenRE decomposes QPP into independent subtasks of predicting relevance of each item in a ranked list to a given query.<n>We predict an item's relevance by using open-source large language models (LLMs) to ensure scientific relevance.
  arXiv  Detail & Related papers  (2024-04-01T09:33:05Z)
• Prompt Consistency for Zero-Shot Task Generalization [118.81196556175797]
  In this paper, we explore methods to utilize unlabeled data to improve zero-shot performance. Specifically, we take advantage of the fact that multiple prompts can be used to specify a single task, and propose to regularize prompt consistency. Our approach outperforms the state-of-the-art zero-shot learner, T0, on 9 out of 11 datasets across 4 NLP tasks by up to 10.6 absolute points in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-29T19:18:37Z)
• Distant finetuning with discourse relations for stance classification [55.131676584455306]
  We propose a new method to extract data with silver labels from raw text to finetune a model for stance classification. We also propose a 3-stage training framework where the noisy level in the data used for finetuning decreases over different stages. Our approach ranks 1st among 26 competing teams in the stance classification track of the NLPCC 2021 shared task Argumentative Text Understanding for AI Debater.
  arXiv  Detail & Related papers  (2022-04-27T04:24:35Z)
• Self-Supervised Relational Reasoning for Representation Learning [5.076419064097733]
  In self-supervised learning, a system is tasked with achieving a surrogate objective by defining alternative targets on unlabeled data. We propose a novel self-supervised formulation of relational reasoning that allows a learner to bootstrap a signal from information implicit in unlabeled data. We evaluate the proposed method following a rigorous experimental procedure, using standard datasets, protocols, and backbones.
  arXiv  Detail & Related papers  (2020-06-10T14:24:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.