SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

• URL: http://arxiv.org/abs/2605.22564v1
• Date: Thu, 21 May 2026 14:45:02 GMT
• Title: SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
• Abstract summary: SynAE is an evaluation framework for assessing how well synthetic benchmarks for tool-calling agents replicate and augment the characteristics of real data trajectories.<n>We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes.
• Score: 18.71623023651951
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Today, tool-calling agents are commonly evaluated or tested on static datasets of execution traces, including input commands, agent responses, and associated tool calls. However, internal production datasets are often insufficient or unusable for testing; for example, they may contain sensitive or proprietary data, or they may be too sparse to support comprehensive testing (especially pre-deployment). In these settings, practitioners are increasingly replacing or augmenting real datasets with synthetic ones for evaluation purposes. A key challenge is quantifying the relation between these synthetic datasets and the real data. We introduce SynAE, an evaluation framework for assessing how well synthetic benchmarks for multi-turn, tool-calling agents replicate and augment the characteristics of real data trajectories. SynAE assesses the validity, fidelity, and diversity of synthetic data across four metric categories: (i) task instructions and intermediate responses, (ii) tool calls, (iii) final outputs, and (iv) downstream evaluation. We evaluate SynAE using recent agent benchmarks and test common synthetic data failure modes via realistic and controlled generation schemes. SynAE detects fine-grained variations in data validity, fidelity and diversity, and shows that no single metric is sufficient to fully characterize synthetic data quality, motivating a multi-axis evaluation of synthetic data for agent testing. A demo of SynAE is available at https://synae-2026-synae-demo.static.hf.space/index.html, with code at https://github.com/wsqwsq/SynAE.

Related papers

• Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets [1.2089037589460674]
  We present Memisis, a tool that orchestrates and evaluates synthetic data.<n>Our tool creates a unified workflow for data generation, validation and evaluation.
  arXiv  Detail & Related papers  (2026-05-18T02:24:52Z)
• Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection [23.019466723490297]
  This work explores the role of synthetic data in improving the detection of Hand-Object Interactions from egocentric images.<n>By using synthetic data and only 10% of the real labeled data, we achieve improvements in Overall AP over models trained exclusively on real data.
  arXiv  Detail & Related papers  (2026-03-31T13:32:37Z)
• Adapting Web Agents with Synthetic Supervision [80.89365133130558]
  Web agents struggle to adapt to new websites due to the scarcity of environment specific tasks and demonstrations.<n>Recent works have explored synthetic data generation to address this challenge.<n>We propose SynthAgent, a fully synthetic supervision framework.
  arXiv  Detail & Related papers  (2025-11-08T18:45:33Z)
• Understanding the Influence of Synthetic Data for Text Embedders [52.04771455432998]
  We first reproduce and publicly release the synthetic data proposed by Wang et al.<n>We critically examine where exactly synthetic data improves model generalization.<n>Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders.
  arXiv  Detail & Related papers  (2025-09-07T19:28:52Z)
• SynDelay: A Synthetic Dataset for Delivery Delay Prediction [50.56729406793283]
  We present SynDelay, a synthetic dataset designed for delivery delay prediction.<n>It is publicly available through the Supply Chain Data Hub, an open initiative promoting dataset sharing and benchmarking in supply chain AI.
  arXiv  Detail & Related papers  (2025-08-30T21:54:37Z)
• Valid Inference with Imperfect Synthetic Data [39.10587411316875]
  We introduce a new estimator based on generalized method of moments.<n>We find that interactions between the moment residuals of synthetic data and those of real data can greatly improve estimates of the target parameter.
  arXiv  Detail & Related papers  (2025-08-08T18:32:52Z)
• Scaling Laws of Synthetic Data for Language Models [125.41600201811417]
  We introduce SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets.<n>Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm.
  arXiv  Detail & Related papers  (2025-03-25T11:07:12Z)
• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [69.28268603137546]
  We show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. This synthetic data is employed to evaluate the robustness of pretrained segmenters. We demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters.
  arXiv  Detail & Related papers  (2023-12-14T18:56:07Z)
• TarGEN: Targeted Data Generation with Large Language Models [51.87504111286201]
  TarGEN is a multi-step prompting strategy for generating high-quality synthetic datasets. We augment TarGEN with a method known as self-correction empowering LLMs to rectify inaccurately labeled instances. A comprehensive analysis of the synthetic dataset compared to the original dataset reveals similar or higher levels of dataset complexity and diversity.
  arXiv  Detail & Related papers  (2023-10-27T03:32:17Z)
• Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark [56.8042116967334]
  Synthetic data serves as an alternative in training machine learning models. ensuring that synthetic data mirrors the complex nuances of real-world data is a challenging task. This paper explores the potential of integrating data-centric AI techniques to guide the synthetic data generation process.
  arXiv  Detail & Related papers  (2023-10-25T20:32:02Z)
• TabSynDex: A Universal Metric for Robust Evaluation of Synthetic Tabular Data [14.900342838726747]
  We propose a new universal metric, TabSynDex, for robust evaluation of synthetic data. Being a single score metric, TabSynDex can also be used to observe and evaluate the training of neural network based approaches.
  arXiv  Detail & Related papers  (2022-07-12T04:08:11Z)
• Synthetic Benchmarks for Scientific Research in Explainable Machine Learning [14.172740234933215]
  We release XAI-Bench: a suite of synthetic datasets and a library for benchmarking feature attribution algorithms. Unlike real-world datasets, synthetic datasets allow the efficient computation of conditional expected values. We demonstrate the power of our library by benchmarking popular explainability techniques across several evaluation metrics and identifying failure modes for popular explainers.
  arXiv  Detail & Related papers  (2021-06-23T17:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.