Rag Performance Prediction for Question Answering

• URL: http://arxiv.org/abs/2604.07985v2
• Date: Wed, 15 Apr 2026 08:44:25 GMT
• Title: Rag Performance Prediction for Question Answering
• Abstract summary: We study the performance of a few pre-retrieval and post-retrieval predictors originally devised for ad hoc retrieval.<n>We also study a few post-generation predictors, one of which is novel to this study and posts the best prediction quality.
• Score: 5.438965251905764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We address the task of predicting the gain of using RAG (retrieval augmented generation) for question answering with respect to not using it. We study the performance of a few pre-retrieval and post-retrieval predictors originally devised for ad hoc retrieval. We also study a few post-generation predictors, one of which is novel to this study and posts the best prediction quality. Our results show that the most effective prediction approach is a novel supervised predictor that explicitly models the semantic relationships among the question, retrieved passages, and the generated answer.

Related papers

• Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation [24.439170886636788]
  Key challenge for improving RAG is to predict both the utility of retrieved documents and the quality of the final answers in terms of correctness and relevance.<n>We define two prediction tasks within RAG: retrieval performance prediction and generation performance prediction.<n>We argue that reader-centric features, such as the LLM's perplexity of the retrieved context conditioned on the input query, can further enhance prediction accuracy.
  arXiv  Detail & Related papers  (2026-01-20T23:59:54Z)
• PROPHET: An Inferable Future Forecasting Benchmark with Causal Intervened Likelihood Estimation [46.3251656496956]
  Recent advances in large language model (LLM)-based systems have shown remarkable potential in forecasting future events.<n>Several benchmarks have been established to evaluate the forecasting capabilities by formalizing the event prediction as a retrieval-augmented generation (RAG) and reasoning task.<n>We introduce a new benchmark, PROPHET, which comprises inferable forecasting questions paired with relevant news for retrieval.
  arXiv  Detail & Related papers  (2025-04-02T08:57:42Z)
• Combining Query Performance Predictors: A Reproducibility Study [6.681467202699048]
  As early as 2009, Hauff et al. [28] explored whether different QPP methods may be combined to improve prediction quality.<n>This study revisits Hauff et al.s work to assess the extent of their findings in the light of new prediction methods, evaluation metrics, and datasets.
  arXiv  Detail & Related papers  (2025-03-31T16:01:58Z)
• Correct after Answer: Enhancing Multi-Span Question Answering with Post-Processing Method [11.794628063040108]
  Multi-Span Question Answering (MSQA) requires models to extract one or multiple answer spans from a given context to answer a question. We propose Answering-Classifying-Correcting (ACC) framework, which employs a post-processing strategy to handle incorrect predictions.
  arXiv  Detail & Related papers  (2024-10-22T08:04:32Z)
• Improved prediction of future user activity in online A/B testing [9.824661943331119]
  In online randomized experiments or A/B tests, accurate predictions of participant inclusion rates are of paramount importance. We present a novel, straightforward, and scalable Bayesian nonparametric approach for predicting the rate at which individuals will be exposed to interventions.
  arXiv  Detail & Related papers  (2024-02-05T17:44:21Z)
• Performative Time-Series Forecasting [64.03865043422597]
  We formalize performative time-series forecasting (PeTS) from a machine-learning perspective.<n>We propose a novel approach, Feature Performative-Shifting (FPS), which leverages the concept of delayed response to anticipate distribution shifts.<n>We conduct comprehensive experiments using multiple time-series models on COVID-19 and traffic forecasting tasks.
  arXiv  Detail & Related papers  (2023-10-09T18:34:29Z)
• When No-Rejection Learning is Consistent for Regression with Rejection [11.244583592648443]
  We study a no-reject learning strategy that uses all the data to learn the prediction. This paper investigates a no-reject learning strategy that uses all the data to learn the prediction.
  arXiv  Detail & Related papers  (2023-07-06T11:43:22Z)
• Prediction-Oriented Bayesian Active Learning [51.426960808684655]
  Expected predictive information gain (EPIG) is an acquisition function that measures information gain in the space of predictions rather than parameters. EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models.
  arXiv  Detail & Related papers  (2023-04-17T10:59:57Z)
• What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience [63.75363908696257]
  computational reinforcement learning seeks to construct an agent's perception of the world through predictions of future sensations. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. We introduce a meta-gradient descent process by which an agent learns what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward.
  arXiv  Detail & Related papers  (2022-06-13T21:31:06Z)
• Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
  We show that pre-trained language models with few examples show strong prediction bias across labels. Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features. These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
  arXiv  Detail & Related papers  (2022-04-17T15:55:18Z)
• Improving Event Duration Prediction via Time-aware Pre-training [90.74988936678723]
  We introduce two effective models for duration prediction. One model predicts the range/unit where the duration value falls in (R-pred); and the other predicts the exact duration value E-pred. Our best model -- E-pred, substantially outperforms previous work, and captures duration information more accurately than R-pred.
  arXiv  Detail & Related papers  (2020-11-05T01:52:11Z)
• Counterfactual Predictions under Runtime Confounding [74.90756694584839]
  We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data. We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
  arXiv  Detail & Related papers  (2020-06-30T15:49:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.