Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I

• URL: http://arxiv.org/abs/2407.02464v1
• Date: Tue, 2 Jul 2024 17:44:00 GMT
• Title: Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I
• Authors: Harrie Oosterhuis, Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky,
• Abstract summary: Large language models (LLMs) can generate relevance annotations at an enormous scale with relatively small computational costs. We propose two methods based on prediction-powered inference and conformal risk control. Our experimental results show that our CIs accurately capture both the variance and bias in evaluation.
• Score: 39.92942310783174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The traditional evaluation of information retrieval (IR) systems is generally very costly as it requires manual relevance annotation from human experts. Recent advancements in generative artificial intelligence -- specifically large language models (LLMs) -- can generate relevance annotations at an enormous scale with relatively small computational costs. Potentially, this could alleviate the costs traditionally associated with IR evaluation and make it applicable to numerous low-resource applications. However, generated relevance annotations are not immune to (systematic) errors, and as a result, directly using them for evaluation produces unreliable results. In this work, we propose two methods based on prediction-powered inference and conformal risk control that utilize computer-generated relevance annotations to place reliable confidence intervals (CIs) around IR evaluation metrics. Our proposed methods require a small number of reliable annotations from which the methods can statistically analyze the errors in the generated annotations. Using this information, we can place CIs around evaluation metrics with strong theoretical guarantees. Unlike existing approaches, our conformal risk control method is specifically designed for ranking metrics and can vary its CIs per query and document. Our experimental results show that our CIs accurately capture both the variance and bias in evaluation based on LLM annotations, better than the typical empirical bootstrapping estimates. We hope our contributions bring reliable evaluation to the many IR applications where this was traditionally infeasible.

Related papers

• Membership Inference Attacks from Causal Principles [24.370456956570873]
  We frame MIA evaluation as a causal inference problem, defining memorization as the causal effect of including a data point in the training set.<n>We propose practical estimators for multi-run, one-run, and zero-run regimes with non-asymptotic consistency guarantees.
  arXiv  Detail & Related papers  (2026-02-02T21:17:28Z)
• Redefining Retrieval Evaluation in the Era of LLMs [20.75884808285362]
  Traditional Information Retrieval (IR) metrics assume that human users sequentially examine documents with diminishing attention to lower ranks.<n>This assumption breaks down in Retrieval Augmented Generation (RAG) systems, where search results are consumed by Large Language Models (LLMs)<n>We introduce a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones.
  arXiv  Detail & Related papers  (2025-10-24T13:17:00Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Probabilistic Runtime Verification, Evaluation and Risk Assessment of Visual Deep Learning Systems [3.9341402479278216]
  We propose a novel methodology for the verification, evaluation, and risk assessment of deep learning systems.<n>Our approach explicitly models the incidence of distributional shifts at runtime by estimating their probability from outputs of out-of-distribution detectors.<n>Our approach consistently outperforms conventional evaluation, with accuracy estimation errors typically ranging between 0.01 and 0.1.
  arXiv  Detail & Related papers  (2025-09-23T16:16:02Z)
• CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection [56.302586730134806]
  We introduce Confidence-Consistency Evaluation (CCE), a novel evaluation metric.<n>CCE simultaneously measures prediction confidence and uncertainty consistency.<n>We also establish RankEval, a benchmark for comparing the ranking capabilities of various metrics.
  arXiv  Detail & Related papers  (2025-09-01T03:38:38Z)
• Accurate Estimation of Mutual Information in High Dimensional Data [0.0]
  Mutual information (MI) is a fundamental measure of statistical dependence between two variables.<n>Recent machine learning-based estimators show promise, but their accuracy depends sensitively on dataset size and structure.<n>We close these gaps through a systematic evaluation of classical and neural MI estimators across standard benchmarks and new synthetic datasets.
  arXiv  Detail & Related papers  (2025-05-31T01:06:18Z)
• Trust, or Don't Predict: Introducing the CWSA Family for Confidence-Aware Model Evaluation [0.0]
  We introduce two new metrics Confidence-Weighted Selective Accuracy (CWSA) and its normalized variant CWSA+.<n>CWSA offers principled and interpretable way to evaluate predictive models under confidence thresholds.<n>We show that CWSA and CWSA+ both effectively detect nuanced failure modes and outperform classical metrics in trust-sensitive tests.
  arXiv  Detail & Related papers  (2025-05-24T10:07:48Z)
• SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts [0.6291443816903801]
  This paper introduces a novel framework designed to autonomously evaluate the robustness of large language models (LLMs) Our method generates descriptive sentences from domain-constrained knowledge graph triplets to formulate adversarial prompts. This self-evaluation mechanism allows the LLM to evaluate its robustness without the need for external benchmarks.
  arXiv  Detail & Related papers  (2024-12-01T10:58:53Z)
• The Certainty Ratio $C_ρ$: a novel metric for assessing the reliability of classifier predictions [0.0]
  This paper introduces the Certainty Ratio ($C_rho$), a novel metric designed to quantify the contribution of confident (certain) versus uncertain predictions to any classification performance measure. Experimental results across 26 datasets and multiple classifiers, including Decision Trees, Naive-Bayes, 3-Nearest Neighbors, and Random Forests, demonstrate that $C_rho$rho reveals critical insights that conventional metrics often overlook.
  arXiv  Detail & Related papers  (2024-11-04T10:50:03Z)
• A practical approach to evaluating the adversarial distance for machine learning classifiers [2.2120851074630177]
  This paper investigates the estimation of the more informative adversarial distance using iterative adversarial attacks and a certification approach. We find that our adversarial attack approach is effective compared to related implementations, while the certification method falls short of expectations.
  arXiv  Detail & Related papers  (2024-09-05T14:57:01Z)
• Evaluating Deep Neural Networks in Deployment (A Comparative and Replicability Study) [11.242083685224554]
  Deep neural networks (DNNs) are increasingly used in safety-critical applications. We study recent approaches that have been proposed to evaluate the reliability of DNNs in deployment. We find that it is hard to run and reproduce the results for these approaches on their replication packages and even more difficult to run them on artifacts other than their own.
  arXiv  Detail & Related papers  (2024-07-11T17:58:12Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Towards Robust and Interpretable EMG-based Hand Gesture Recognition using Deep Metric Meta Learning [37.21211404608413]
  We propose a shift to deep metric-based meta-learning in EMG PR to supervise the creation of meaningful and interpretable representations. We derive a robust class proximity-based confidence estimator that leads to a better rejection of incorrect decisions.
  arXiv  Detail & Related papers  (2024-04-17T23:37:50Z)
• TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness [58.721012475577716]
  Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, prompting a surge in their practical applications. This paper introduces TrustScore, a framework based on the concept of Behavioral Consistency, which evaluates whether an LLMs response aligns with its intrinsic knowledge.
  arXiv  Detail & Related papers  (2024-02-19T21:12:14Z)
• TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR [1.8477401359673709]
  Class-probability-based confidence scores do not accurately represent quality of overconfident ASR predictions. We propose a novel Temporal-Lexeme Similarity (TeLeS) confidence score to train Confidence Estimation Model (CEM) We conduct experiments with ASR models trained in three languages, namely Hindi, Tamil, and Kannada, with varying training data sizes.
  arXiv  Detail & Related papers  (2024-01-06T16:29:13Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation [63.46331073232526]
  We present an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric.
  arXiv  Detail & Related papers  (2021-05-30T10:04:13Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.