Related papers: What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations

URL: http://arxiv.org/abs/2608.17719v1
Date: Tue, 18 Aug 2026 12:44:54 GMT
Title: What Aggregate Scores Miss: Measuring Item-Level Regressions in Commercial LLM API Migrations
Abstract summary: Software systems that depend on large language model APIs must migrate to successor versions when vendors deprecate older models.<n>Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure.
Score: 1.6787220460106658
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Context: Software systems that depend on commercial large language model APIs must migrate to successor versions when vendors deprecate older models. Migration decisions typically rely on aggregate benchmark scores, which compress heterogeneous item-level behaviour into a single net figure. Objective: We measure what that compression conceals. Method: On three pairwise upgrades in the GPT-5.4 to GPT-5.6 Sol product sequence, we query 900 public benchmark items (graduate-level knowledge, olympiad mathematics, instruction following) 50 times per item per model, classify each item as reliably improved, reliably regressed, practically equivalent, or inconclusive under false-discovery-rate control and a practical-significance threshold, and calibrate the results against a label-permutation null. Results: Across all nine migration-benchmark cells, reliable improvements and reliable regressions coexist. Edges with aggregate gains of up to 7.3 percentage points contain up to 8.3% reliably regressed items; edges with aggregate losses contain up to 10.7% reliably improved items. On the instruction-following benchmark, the gap between strict and loose scoring widens by 3.9 percentage points on the latest migration: a 3.9-point regression under strict scoring shrinks to 0.04 points under loose scoring. Conclusion: Migration decisions based on aggregate scores alone miss substantial bidirectional item-level change. The complete response-level archive and per-item scoring outputs are released.

Related papers

ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training [66.72708893922605]
Post-training of large multimodal models should add new capabilities while preserving those from pre-training.<n>SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting.<n>We introduce ReDraft, which obtains both from the model's own failures.
arXiv Detail & Related papers (2026-09-15T04:59:03Z)
Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence [0.7340017786387767]
We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values.<n>On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label.
arXiv Detail & Related papers (2026-08-26T14:41:24Z)
AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets [4.193537335690018]
We present AgentFAIR, a multi-agent framework combining structured extraction with sub-principle-specific evaluators.<n>Each produces a 0-3 maturity score, cited evidence, and recommendations.<n>A preliminary 15-dataset expert study yields Fleiss' kappa of 0.71 and 82% alignment with expert consensus.
arXiv Detail & Related papers (2026-07-17T09:32:54Z)
Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs [11.997694190254974]
Practice often treats automatic metrics for attribution as interchangeable.<n>We audit eight automatic scorers across three evaluation constructs.<n>A naive "best-on-average" rule for choosing an evaluator fails leave-one-dataset-out.
arXiv Detail & Related papers (2026-06-22T20:25:36Z)
Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
arXiv Detail & Related papers (2026-04-20T00:57:31Z)
Calibrated Confidence Estimation for Tabular Question Answering [0.0]
All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA)<n>A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models.<n>A secondary contribution, structure-aware recalibration, improves AUROC by percentage points over standard post-hoc methods.
arXiv Detail & Related papers (2026-04-14T09:16:53Z)
DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles [5.647839536820347]
We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement to produce well-calibrated confidence estimates.<n>DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline.<n>The learned features generalize across benchmarks with near-zero performance degradation.
arXiv Detail & Related papers (2026-03-21T23:24:12Z)
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
arXiv Detail & Related papers (2026-03-17T21:19:08Z)
CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
arXiv Detail & Related papers (2025-12-22T16:34:21Z)
Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
arXiv Detail & Related papers (2025-09-25T14:05:55Z)
MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning [80.15393178083607]
Large Language Models' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples and voting among generated samples.<n> Refinement offers an alternative by using LLM-generated feedback to improve solution quality.<n>We propose MAgICoRe, which avoids excessive refinement by categorizing problem difficulty as easy or hard.
arXiv Detail & Related papers (2024-09-18T17:12:41Z)
LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
arXiv Detail & Related papers (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.