What Is Missing: Interpretable Ratings for Large Language Model Outputs

• URL: http://arxiv.org/abs/2603.04429v1
• Date: Tue, 17 Feb 2026 14:04:42 GMT
• Title: What Is Missing: Interpretable Ratings for Large Language Model Outputs
• Abstract summary: We introduce the What Is Missing (WIM) rating system to produce rankings from natural-language feedback.<n>WIM integrates into existing training pipelines, can be combined with other rating techniques, and can be used as input to any preference learning method.<n>We empirically observe that, compared to discrete numerical ratings, WIM yields fewer ties and larger rating deltas.
• Score: 4.402604078675521
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current Large Language Model (LLM) preference learning methods such as Proximal Policy Optimization and Direct Preference Optimization learn from direct rankings or numerical ratings of model outputs, these rankings are subjective, and a single numerical rating chosen directly by a judge is a poor proxy for the quality of natural language, we introduce the What Is Missing (WIM) rating system to produce rankings from natural-language feedback, WIM integrates into existing training pipelines, can be combined with other rating techniques, and can be used as input to any preference learning method without changing the learning algorithm, to compute a WIM rating, a human or LLM judge writes feedback describing what the model output is missing, we embed the output and the feedback with a sentence embedding model and compute the cosine similarity between the resulting vectors, we empirically observe that, compared to discrete numerical ratings, WIM yields fewer ties and larger rating deltas, which improves the availability of a learning signal in pairwise preference data, we use interpretable in the following limited sense: for each scalar rating, we can inspect the judge's missing-information text that produced it, enabling qualitative debugging of the preference labels.

Related papers

• Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains [67.71020482405343]
  We study how to design algorithms that can leverage additional information in the form of rating gap.<n>We present new algorithms that can achieve faster statistical rates than DPO in presence of accurate rating gap information.
  arXiv  Detail & Related papers  (2026-01-31T08:38:21Z)
• Intuitionistic Fuzzy Sets for Large Language Model Data Annotation: A Novel Approach to Side-by-Side Preference Labeling [0.0]
  This paper introduces a novel framework based on intuitionistic fuzzy sets (IFS) for modeling and aggregating human preferences in large language models (LLMs)<n>Our approach captures not only the degree of preference but also the uncertainty and hesitation inherent in human judgment through membership, non-membership, and hesitation degrees.<n> Experimental validation on multiple datasets demonstrates that our IFS-based approach significantly improves annotation consistency, reduces annotator fatigue, and produces higher-quality preference data.
  arXiv  Detail & Related papers  (2025-05-30T04:20:00Z)
• Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF [67.48004037550064]
  We propose an active learning approach to efficiently select prompt and preference pairs.<n>Our method evaluates the gradients of all potential preference annotations to assess their impact on model updates.<n> Experimental results demonstrate that our method outperforms the baseline by up to 5% in win rates against the chosen completion.
  arXiv  Detail & Related papers  (2025-03-28T04:22:53Z)
• MAPLE: A Framework for Active Preference Learning Guided by Large Language Models [9.37268652939886]
  We introduce MAPLE, a framework for large language model-guided Bayesian active preference learning.<n>Our results demonstrate that MAPLE accelerates the learning process and effectively improves humans' ability to answer queries.
  arXiv  Detail & Related papers  (2024-12-10T05:55:14Z)
• Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback [87.37721254914476]
  We introduce HyPER, a Hybrid Preference routER that defers an annotation to either humans or language models (LMs)<n>We show that the selected hybrid mixture of synthetic and direct human preferences using HyPER achieves better RM performance compared to using either one exclusively by 7-13% on RewardBench.<n>We also analyze features from HyPER and find that prompts with moderate safety concerns or complexity benefit the most from human feedback.
  arXiv  Detail & Related papers  (2024-10-24T20:04:15Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Calibrating LLM-Based Evaluator [92.17397504834825]
  We propose AutoCalibrate, a multi-stage, gradient-free approach to calibrate and align an LLM-based evaluator toward human preference. Instead of explicitly modeling human preferences, we first implicitly encompass them within a set of human labels. Our experiments on multiple text quality evaluation datasets illustrate a significant improvement in correlation with expert evaluation through calibration.
  arXiv  Detail & Related papers  (2023-09-23T08:46:11Z)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [69.91340332545094]
  We introduce FLASK, a fine-grained evaluation protocol for both human-based and model-based evaluation. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance.
  arXiv  Detail & Related papers  (2023-07-20T14:56:35Z)
• Training Language Models with Language Feedback at Scale [50.70091340506957]
  We introduce learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback.
  arXiv  Detail & Related papers  (2023-03-28T17:04:15Z)
• Evaluation of HTR models without Ground Truth Material [2.4792948967354236]
  evaluation of Handwritten Text Recognition models during their development is straightforward. But the evaluation process becomes tricky as soon as we switch from development to application. We show that lexicon-based evaluation can compete with lexicon-based methods.
  arXiv  Detail & Related papers  (2022-01-17T01:26:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.