Data and Evaluation Closed-Loop for Model Capability Enhancement

• URL: http://arxiv.org/abs/2606.28471v1
• Date: Fri, 26 Jun 2026 14:45:57 GMT
• Title: Data and Evaluation Closed-Loop for Model Capability Enhancement
• Abstract summary: We show that evaluation can be routine, auditable, and experimentally validated rather than intuitive.<n>We show that the evaluation-to-data inference can be routine, auditable, and experimentally validated rather than intuitive.
• Score: 8.245706308161504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it only retrospectively, compressing samples, prompts, decoding, and scoring rules into one noisy score. Practical optimization runs this backward: a failure is observed first, and the engineer must infer the corpus fix. The two sides speak incompatible vocabularies -- benchmark names and per-sample correctness versus data sources, domains, and quality labels -- so this inference is usually intuition, not method. We close this gap with the \emph{capability slice}: a group of evaluation samples sharing background condition, task type, solving operation, and output constraint -- precise enough to localize a single weakness yet stable enough to survive aggregation, unlike a benchmark name, too coarse, or a single sample, too noisy. Built around this unit, an evaluation taxonomy, a non-instruction data taxonomy, and mapping rules form a closed loop turning a benchmark-level failure into a targeted, testable data intervention. We test this loop on two case studies pulling in opposite directions. First, the loop rules the data out: continued pre-training drives BBH down by $-46.82\%$, but diagnosis traces this to a single masked \texttt{\textless EOS\textgreater} loss rather than weakened reasoning; restoring it recovers BBH to $66.44$, above the original checkpoint, without changing the data. Second, the loop rules the data in: a persistent math-reasoning weakness is decomposed by solving operation into specific failing combinations, and a weakness-targeted sampling procedure built from it lifts AIME2025/AIME2026 Pass@128 from $6.67$/$0.00$ to $26.67$ each. The same unmodified loop reaches opposite, correct verdicts in both cases, showing the evaluation-to-data inference can be routine, auditable, and experimentally validated rather than intuitive.

Related papers

• When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide [0.0]
  We benchmark six estimators across five datasets and two known-effect sweeps, and validate the mechanisms against a non-simulated paired reference.<n>Sharpening a logger from the target's own score barely moves overlap over the tested range; action-level disagreement collapses it.<n>Cross-fitting the outcome nuisance alone leaves the reuse bias in place and makes it worse.
  arXiv  Detail & Related papers  (2026-08-12T18:10:10Z)
• Accuracy and Order Sensitivity Diverge Under Label-Free Strategies [4.127762845164668]
  Multiple-choice benchmarks are widely used to evaluate large language models.<n>Multiple choice scores conflate knowledge with sensitivity to option order.<n>In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence.
  arXiv  Detail & Related papers  (2026-08-12T11:34:45Z)
• Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation [2.733105115068754]
  We introduce a scalable, inference-only data valuation pipeline that approximates the Shapley value without iterative model retraining.<n>We demonstrate the pipeline's efficacy in sanitizing two heavily vetted alignment datasets.
  arXiv  Detail & Related papers  (2026-07-24T03:31:44Z)
• The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval [21.243421703047037]
  We study how word-boundary corruption affects how Large Language Model benchmarks detect targeted information.<n>By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate.
  arXiv  Detail & Related papers  (2026-05-08T03:26:42Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• Don't Pass$\mathtt{@}k$: A Bayesian Framework for Large Language Model Evaluation [4.082208996639461]
  Pass$@k$ is widely used to report performance for LLM reasoning, but it often yields unstable, misleading rankings.<n>We present a principled Bayesian evaluation framework that replaces Pass$@k$ with posterior estimates of a model's underlying success probability and credible intervals.
  arXiv  Detail & Related papers  (2025-10-05T16:14:03Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Training on the Benchmark Is Not All You Need [52.01920740114261]
  We propose a simple and effective data leakage detection method based on the contents of multiple-choice options.<n>Our method is able to work under gray-box conditions without access to model training data or weights.<n>We evaluate the degree of data leakage of 35 mainstream open-source LLMs on four benchmark datasets.
  arXiv  Detail & Related papers  (2024-09-03T11:09:44Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models [25.022166664832596]
  We propose a simple and effective approach to detect data contamination in large language models (LLMs)<n>We frame data contamination detection as a series of multiple-choice questions, devising a quiz format wherein three perturbed versions of each instance, subsampled from a specific dataset partition, are created.<n>The generated perturbations, along with the original dataset instance, form the options in the DCQ, with an extra option accommodating the selection of none of the provided options.
  arXiv  Detail & Related papers  (2023-11-10T18:48:58Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Bias Mimicking: A Simple Sampling Approach for Bias Mitigation [57.17709477668213]
  We introduce a new class-conditioned sampling method: Bias Mimicking. Bias Mimicking improves underrepresented groups' accuracy of sampling methods by 3% over four benchmarks.
  arXiv  Detail & Related papers  (2022-09-30T17:33:00Z)
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation [109.06060143938052]
  We propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English.
  arXiv  Detail & Related papers  (2021-04-12T06:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.