Risk-Controlled Selective LLM Answering by Pricing Label-Free Checks
Abstract Overview
This paper studies selective answering for LLMs under explicit risk control, focusing on how to decide when to serve an answer and when to abstain. It introduces PriceCheck, which represents label-free checks such as re-solving and backward probing by their agreement rates on correct versus incorrect answers together with their per-run cost. These fitted prices are composed into a small family of schedules, including cascades and adaptive stopping rules, and a calibration test then selects a schedule at a stated selective-risk target. The evaluation is conducted on a frozen pool of mathematics answers from five fine-tuned generators under a shared certification protocol against several verifier-style baselines.
Novelty
The distinctive idea is to treat label-free verification checks as priced actions characterized by completeness, leak, and cost, then compose them into selective-answering schedules before certification. Rather than relying only on a verifier score or ranking quality, the method explicitly optimizes how checks are ordered, combined, and stopped under a finite-family risk-control framework.
Results
At the primary 1.5% selective-risk target, the selected PriceCheck schedules serve 76.1% of answers on average and keep held-out selective risk below the target on all 15 splits. Under the shared protocol, it serves more answers at that target than reward models, a prompted judge, generator confidence, and a trained correctness classifier, and it retains the fewest wrong answers at matched coverage. Across 118 diagnostic schedules, price-based coverage predictions achieve a rank correlation of 0.97 with observed coverage.
Key Points
- PriceCheck models each label-free check using agreement on correct answers, agreement on wrong answers, and execution cost, then builds selective-answering schedules from these quantities.
- The method uses calibration-time certification over a fixed schedule family to meet stated selective-risk targets, instead of depending only on a scalar verifier score.
- Empirically, the approach achieves stronger certified coverage-risk tradeoffs than multiple baselines on mathematics answer selection, while accurately ranking schedule coverage across a large diagnostic set.