NodeSynth: Socially Aligned Synthetic Data for AI Evaluation

• URL: http://arxiv.org/abs/2605.14381v2
• Date: Sat, 16 May 2026 23:33:47 GMT
• Title: NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
• Abstract summary: We introduce Node Synth, an evidence-grounded methodology that generates socially relevant synthetic queries.<n>Node Synth elicited failure rates up to five times higher than human-authored benchmarks.<n>We open-source our end-to-end research prototype and datasets to enable scalable, high-stakes model evaluation.
• Score: 13.768105966930916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSynth, an evidence-grounded methodology that generates socially relevant synthetic queries by leveraging a fine-tuned taxonomy generator (TaG) anchored in real-world evidence. Evaluated against four mainstream LLMs (e.g., Claude 4.5 Haiku), NodeSynth elicited failure rates up to five times higher than human-authored benchmarks. Ablation studies confirm that our granular taxonomic expansion significantly drives these failure rates, while independent validation reveals critical deficiencies in prominent guard models (e.g., Llama-Guard-3). We open-source our end-to-end research prototype and datasets to enable scalable, high-stakes model evaluation and targeted safety interventions (https://github.com/google-research/nodesynth).

Related papers

• Reinforcement-Guided Synthetic Data Generation for Privacy-Sensitive Identity Recognition [60.52810518437911]
  High-fidelity generative models are increasingly needed in privacy-sensitive scenarios.<n> limited data leads to poor generative models, which in turn fail to mitigate data scarcity.<n>We propose a reinforcement-guided synthetic data generation framework that adapts general-domain generative priors to identity recognition tasks.
  arXiv  Detail & Related papers  (2026-04-09T06:52:03Z)
• Reasoning-Driven Synthetic Data Generation and Evaluation [3.4565140548638884]
  Simula is a novel reasoning-driven framework for data generation and evaluation.<n>It employs a seedless, agentic approach to generate synthetic datasets at scale.<n>We show the efficacy of our approach on a variety of datasets, rigorously testing both intrinsic and downstream properties.
  arXiv  Detail & Related papers  (2026-03-31T14:26:33Z)
• Understanding the Influence of Synthetic Data for Text Embedders [52.04771455432998]
  We first reproduce and publicly release the synthetic data proposed by Wang et al.<n>We critically examine where exactly synthetic data improves model generalization.<n>Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders.
  arXiv  Detail & Related papers  (2025-09-07T19:28:52Z)
• SynDelay: A Synthetic Dataset for Delivery Delay Prediction [50.56729406793283]
  We present SynDelay, a synthetic dataset designed for delivery delay prediction.<n>It is publicly available through the Supply Chain Data Hub, an open initiative promoting dataset sharing and benchmarking in supply chain AI.
  arXiv  Detail & Related papers  (2025-08-30T21:54:37Z)
• Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Era [49.46005489386284]
  This tutorial introduces the foundations and latest advances in synthetic data generation.<n> Attendees will gain actionable insights into leveraging generative synthetic data to enhance data mining research and practice.
  arXiv  Detail & Related papers  (2025-08-27T05:04:07Z)
• Valid Inference with Imperfect Synthetic Data [39.10587411316875]
  We introduce a new estimator based on generalized method of moments.<n>We find that interactions between the moment residuals of synthetic data and those of real data can greatly improve estimates of the target parameter.
  arXiv  Detail & Related papers  (2025-08-08T18:32:52Z)
• How Good Are Synthetic Requirements ? Evaluating LLM-Generated Datasets for AI4RE [0.5156484100374059]
  This paper presents an enhanced Product Line approach for generating synthetic requirements data.<n>We investigate four research questions assessing how prompting strategies, automated prompt optimization, and post-generation curation affect data quality.<n>Our results show that synthetic requirements can match or outperform human-authored ones for specific tasks.
  arXiv  Detail & Related papers  (2025-06-26T10:52:07Z)
• AugGen: Synthetic Augmentation using Diffusion Models Can Improve Recognition [14.525986333650417]
  Synthetic data generation offers a promising alternative to external datasets or pre-trained models.<n>AugGen strategically samples from a class-conditional generative model trained exclusively on the target FR dataset.<n>Our findings demonstrate that carefully integrated synthetic data can both mitigate privacy constraints and substantially enhance recognition performance.
  arXiv  Detail & Related papers  (2025-03-14T16:10:21Z)
• Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark [56.8042116967334]
  Synthetic data serves as an alternative in training machine learning models. ensuring that synthetic data mirrors the complex nuances of real-world data is a challenging task. This paper explores the potential of integrating data-centric AI techniques to guide the synthetic data generation process.
  arXiv  Detail & Related papers  (2023-10-25T20:32:02Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Hybrid Deep Learning Model using SPCAGAN Augmentation for Insider Threat Analysis [7.576808824987132]
  Anomaly detection using deep learning requires comprehensive data, but insider threat data is not readily available due to confidentiality concerns. We propose a linear manifold learning-based generative adversarial network, SPCAGAN, that takes input from heterogeneous data sources. We show that our proposed approach has a lower error, is more accurate, and generates substantially superior synthetic insider threat data than previous models.
  arXiv  Detail & Related papers  (2022-03-06T02:08:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.