A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance

• URL: http://arxiv.org/abs/2609.16592v1
• Date: Tue, 15 Sep 2026 03:36:35 GMT
• Title: A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance
• Abstract summary: This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets.<n>We introduce a schema eliciting key information about the goals, scope, and context of an evaluation task, and use this information to guide synthetic data generation.<n>We show how expert-informed scaffolds can guide synthetic data generation toward more valid benchmarks.
• Score: 8.655727081754241
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with synthetic data generation. Existing benchmark construction methods often trade off validity and scalability: datasets designed with domain experts can produce high-quality evaluations but are slow and costly to create, while synthetically generating data may scale efficiently but often results in unrealistic, redundant, or out-of-scope examples. To address this gap, we introduce a schema eliciting key information about the goals, scope, and context of an evaluation task, and use this information to guide synthetic data generation. We further define four criteria grounded in measurement validity for assessing dataset quality: coverage, diversity, content realism, and stylistic realism. Using these criteria, we show how expert-informed scaffolds can guide synthetic data generation toward more valid benchmarks. Through quantitative evaluations and a real-world case study with domain experts, we demonstrate that our approach improves benchmark data quality over existing methods while preserving validity. We additionally analyze how different types of schema information affect different dataset quality criteria, and provide practical guidance on which information to prioritize collecting under resource constraints.

Related papers

• A Benchmarking Framework for Model Datasets [1.2234742322758418]
  Empirical and LLM-based research in model-driven engineering increasingly relies on datasets of software models.<n>Such datasets are typically collected or created ad hoc and without guarantees of their quality for the specific task for which they are used.<n>We propose a Benchmark Platform for MDE that provides a unified infrastructure for systematically assessing and comparing datasets of software models across languages and formats.
  arXiv  Detail & Related papers  (2026-03-05T15:04:35Z)
• It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks [87.7937890373758]
  Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.<n>We introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks.<n>We propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels.
  arXiv  Detail & Related papers  (2026-02-12T16:31:01Z)
• The LLM Data Auditor: A Metric-oriented Survey on Quality and Trustworthiness in Evaluating Synthetic Data [25.926467401802046]
  Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.<n>We propose a framework for evaluating synthetic data from two dimensions: quality and trustworthiness.
  arXiv  Detail & Related papers  (2026-01-25T06:40:25Z)
• OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
  OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
  arXiv  Detail & Related papers  (2025-12-16T03:33:24Z)
• Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability [41.23032741638842]
  This position paper advocates for the integration of systematic, descriptive-based evaluation metrics into the dataset review process.<n>We introduce DataRubrics, a structured framework for assessing the quality of both human- and model-generated datasets.
  arXiv  Detail & Related papers  (2025-06-02T15:31:52Z)
• Benchmarking Synthetic Tabular Data: A Multi-Dimensional Evaluation Framework [0.4874819476581695]
  evaluating the quality of synthetic data remains a key challenge for ensuring privacy and utility in data-driven research.<n>We present a framework that quantifies how well synthetic data replicates original distributional properties while ensuring privacy.
  arXiv  Detail & Related papers  (2025-04-02T17:10:30Z)
• A Survey on Tabular Data Generation: Utility, Alignment, Fidelity, Privacy, and Beyond [53.56796220109518]
  Different use cases demand synthetic data to comply with different requirements to be useful in practice.<n>Four types of requirements are reviewed: utility of the synthetic data, alignment of the synthetic data with domain-specific knowledge, statistical fidelity of the synthetic data distribution compared to the real data distribution, and privacy-preserving capabilities.<n>We discuss future directions for the field, along with opportunities to improve the current evaluation methods.
  arXiv  Detail & Related papers  (2025-03-07T21:47:11Z)
• A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models [3.672850225066168]
  generative AI and large language models (LLMs) have opened up new avenues for producing synthetic data.<n>Despite the potential benefits, concerns regarding privacy leakage have surfaced.<n>We introduce SynEval, an open-source evaluation framework designed to assess the fidelity, utility, and privacy preservation of synthetically generated tabular data.
  arXiv  Detail & Related papers  (2024-04-20T08:08:28Z)
• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [69.28268603137546]
  We show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. This synthetic data is employed to evaluate the robustness of pretrained segmenters. We demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters.
  arXiv  Detail & Related papers  (2023-12-14T18:56:07Z)
• Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark [56.8042116967334]
  Synthetic data serves as an alternative in training machine learning models. ensuring that synthetic data mirrors the complex nuances of real-world data is a challenging task. This paper explores the potential of integrating data-centric AI techniques to guide the synthetic data generation process.
  arXiv  Detail & Related papers  (2023-10-25T20:32:02Z)
• Analyzing Dataset Annotation Quality Management in the Wild [63.07224587146207]
  Even popular datasets used to train and evaluate state-of-the-art models contain a non-negligible amount of erroneous annotations, biases, or artifacts. While practices and guidelines regarding dataset creation projects exist, large-scale analysis has yet to be performed on how quality management is conducted.
  arXiv  Detail & Related papers  (2023-07-16T21:22:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.