Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation

• URL: http://arxiv.org/abs/2605.31278v2
• Date: Thu, 04 Jun 2026 07:25:36 GMT
• Title: Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
• Abstract summary: Prediction-powered inference (PPI) combines both into debiased estimates with valid confidence intervals.<n>We introduce GLIDE, an open-source Python library that unifies state-of-the-art PPI estimators.<n> GLIDE ships with a reproducible Monte Carlo validation suite, an empirically grounded decision tree for method selection, and an agentic evaluation case study.
• Score: 2.28126966226433
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable evaluation of agentic systems requires unbiased estimates with valid uncertainty, but standard practice navigates between costly human annotation and biased LLM-as-judge proxies. Prediction-powered inference (PPI) combines both into debiased estimates with valid confidence intervals, yet its various methods remain scattered across papers under partial implementations. We introduce GLIDE, an open-source Python library that unifies state-of-the-art PPI estimators (PPI++, Stratified PPI, Predict-Then-Debias and its stratified variants, Active Statistical Inference) and samplers (uniform, stratified, active, cost-optimal) under a scipy-style API specialized to mean estimation. GLIDE ships with a reproducible Monte Carlo validation suite, an empirically grounded decision tree for method selection, and an agentic evaluation case study showing substantial annotation savings at equivalent precision. The GLIDE package is available at this URL: https://github.com/EmertonData/glide

Related papers

• Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking [0.20305676256390937]
  This paper decomposes uncertainty into its sources, distinguishes variance that shrinks with more data, and uses design-study projections to reduce total error.<n>Applying the approach to ideology annotation, safety classification, MMLU benchmarking, and a human-validated propaganda audit reveals different dominant variance sources by domain and scoring method.
  arXiv  Detail & Related papers  (2026-04-13T14:58:15Z)
• Demystifying Prediction Powered Inference [4.962232906170314]
  Prediction-Powered Inference (PPI) offers a principled framework that leverages predictions from large unlabeled datasets to improve statistical efficiency.<n>Despite its potential, the growing PPI variants and the subtle distinctions between them have made it challenging for practitioners to determine when and how to apply these methods responsibly.<n>This paper demystifies PPI by synthesizing its theoretical foundations, methodological extensions, connections to existing statistics literature, and diagnostic tools into a unified practical workflow.
  arXiv  Detail & Related papers  (2026-01-28T18:16:02Z)
• Optimal Debiased Inference on Privatized Data via Indirect Estimation and Parametric Bootstrap [12.65121513620053]
  Existing usage of the parametric bootstrap on privatized data ignored or avoided handling the effect of clamping.<n>We propose using the indirect inference method to estimate the parameter values consistently.<n>Our framework produces confidence intervals with well-calibrated coverage and performs hypothesis testing with the correct type I error.
  arXiv  Detail & Related papers  (2025-07-14T19:12:16Z)
• Bayesian Estimation and Tuning-Free Rank Detection for Probability Mass Function Tensors [17.640500920466984]
  This paper presents a novel framework for estimating the joint PMF and automatically inferring its rank from observed data. We derive a deterministic solution based on variational inference (VI) to approximate the posterior distributions of various model parameters. Additionally, we develop a scalable version of the VI-based approach by leveraging variational inference (SVI) Experiments involving both synthetic data and real movie recommendation data illustrate the advantages of our VI and SVI-based methods in terms of estimation accuracy, automatic rank detection, and computational efficiency.
  arXiv  Detail & Related papers  (2024-10-08T20:07:49Z)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
  We introduce the first formal probabilistic evaluation framework for Large Language Models (LLMs)<n> Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model.<n>Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
  arXiv  Detail & Related papers  (2024-10-04T15:44:23Z)
• Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.<n>We propose a method called Stratified Prediction-Powered Inference (StratPPI)<n>We show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies.
  arXiv  Detail & Related papers  (2024-06-06T17:37:39Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Uncertainty-Aware Instance Reweighting for Off-Policy Learning [63.31923483172859]
  We propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator.
  arXiv  Detail & Related papers  (2023-03-11T11:42:26Z)
• MAPIE: an open-source library for distribution-free uncertainty quantification [0.0]
  We introduce MAPIE, an open-source Python library that quantifies the uncertainties of Machine Learning models. MAPIE implements conformgnostical prediction methods, allowing the user to easily compute uncertainties. It is hosted on scikit-learn-contrib and is fully "scikit-learn-compatible"
  arXiv  Detail & Related papers  (2022-07-25T15:44:19Z)
• Control Variates for Slate Off-Policy Evaluation [112.35528337130118]
  We study the problem of off-policy evaluation from batched contextual bandit data with multidimensional actions. We obtain new estimators with risk improvement guarantees over both the PI and self-normalized PI estimators.
  arXiv  Detail & Related papers  (2021-06-15T06:59:53Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.