RPRA: Predicting an LLM-Judge for Efficient but Performant Inference

• URL: http://arxiv.org/abs/2604.12634v1
• Date: Tue, 14 Apr 2026 12:04:21 GMT
• Title: RPRA: Predicting an LLM-Judge for Efficient but Performant Inference
• Abstract summary: Large language models (LLMs) face a fundamental trade-off between computational efficiency (e.g., number of parameters) and output quality.<n>We evaluate three approaches: zero-shot prediction, prediction using an in-context report card, and supervised fine-tuning.
• Score: 46.77119317314821
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) face a fundamental trade-off between computational efficiency (e.g., number of parameters) and output quality, especially when deployed on computationally limited devices such as phones or laptops. One way to address this challenge is by following the example of humans and have models ask for help when they believe they are incapable of solving a problem on their own; we can overcome this trade-off by allowing smaller models to respond to queries when they believe they can provide good responses, and deferring to larger models when they do not believe they can. To this end, in this paper, we investigate the viability of Predict-Answer/Act (PA) and Reason-Predict-Reason-Answer/Act (RPRA) paradigms where models predict -- prior to responding -- how an LLM judge would score their output. We evaluate three approaches: zero-shot prediction, prediction using an in-context report card, and supervised fine-tuning. Our results show that larger models (particularly reasoning models) perform well when predicting generic LLM judges zero-shot, while smaller models can reliably predict such judges well after being fine-tuned or provided with an in-context report card. Altogether, both approaches can substantially improve the prediction accuracy of smaller models, with report cards and fine-tuning achieving mean improvements of up to 55% and 52% across datasets, respectively. These findings suggest that models can learn to predict their own performance limitations, paving the way for more efficient and self-aware AI systems.

Related papers

• Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization [17.34147279018477]
  We propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity.<n>We derive doubly robust estimators of each model's performance that use the low-rank predictions to reduce variance.<n> Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost.
  arXiv  Detail & Related papers  (2026-05-11T11:43:28Z)
• Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference [10.009730627424629]
  Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks.<n>We propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates.
  arXiv  Detail & Related papers  (2026-02-25T16:38:03Z)
• Enhanced Multi-model Online Conformal Prediction [25.495949162960624]
  Conformal prediction is a framework for uncertainty quantification that constructs prediction sets for previously unseen data.<n>The efficiency of these prediction sets, measured by their size, depends on the choice of the underlying learning model.<n>This work develops a novel multi-model online conformal prediction algorithm that reduces computational complexity and improves prediction efficiency.
  arXiv  Detail & Related papers  (2026-01-04T23:44:43Z)
• Using LLMs to Directly Guess Conditional Expectations Can Improve Efficiency in Causal Estimation [0.3222802562733787]
  We show that predictions made by generative models trained on historical data can be used to improve the performance of these estimators.<n>We consider a case study using a small dataset of online jewelry auctions, and demonstrate that inclusion of LLM-generated guesses as predictors can improve efficiency in estimation.
  arXiv  Detail & Related papers  (2025-10-09T03:34:06Z)
• Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading [0.5825410941577593]
  Fine-tuning methods have historically required large-scale compute clusters inaccessible to most users.<n>New closed-model approaches such as OpenAI's fine-tuning service promise results with as few as 100 examples.<n>We evaluate both of these fine-tuning methods, measuring their interaction with few-shot prompting for automated short answer grading.
  arXiv  Detail & Related papers  (2025-08-06T03:52:55Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step.<n>Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy.<n>We propose Reasoning-Driven Process Reward Modeling (R-PRM)<n>R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• Large Language Models Must Be Taught to Know What They Don't Know [97.90008709512921]
  We show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead.<n>We also investigate the mechanisms that enable reliable uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators.
  arXiv  Detail & Related papers  (2024-06-12T16:41:31Z)
• Predictable MDP Abstraction for Unsupervised Model-Based RL [93.91375268580806]
  We propose predictable MDP abstraction (PMA) Instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches.
  arXiv  Detail & Related papers  (2023-02-08T07:37:51Z)
• Predicting on the Edge: Identifying Where a Larger Model Does Better [61.793778186198864]
  We show that large models have the largest improvement on examples where the small model is most uncertain. We show that a switcher model which defers examples to a larger model when a small model is uncertain can achieve striking improvements in performance and resource usage.
  arXiv  Detail & Related papers  (2022-02-15T18:53:14Z)
• Bayes DistNet -- A Robust Neural Network for Algorithm Runtime Distribution Predictions [1.8275108630751844]
  Randomized algorithms are used in many state-of-the-art solvers for constraint satisfaction problems (CSP) and Boolean satisfiability (SAT) problems. Previous state-of-the-art methods directly try to predict a fixed parametric distribution that the input instance follows. This new model achieves robust predictive performance in the low observation setting, as well as handling censored observations.
  arXiv  Detail & Related papers  (2020-12-14T01:15:39Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.