Struct-Bench: A Benchmark for Differentially Private Structured Text Generation

• URL: http://arxiv.org/abs/2509.10696v1
• Date: Fri, 12 Sep 2025 21:18:13 GMT
• Title: Struct-Bench: A Benchmark for Differentially Private Structured Text Generation
• Abstract summary: Struct-Bench is a framework for evaluating synthetic datasets derived from structured datasets that contain natural language data.<n>Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with Context-Free Grammars (CFGs)<n>We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods.
• Score: 34.596107504223944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Differentially private (DP) synthetic data generation is a promising technique for utilizing private datasets that otherwise cannot be exposed for model training or other analytics. While much research literature has focused on generating private unstructured text and image data, in enterprise settings, structured data (e.g., tabular) is more common, often including natural language fields or components. Existing synthetic data evaluation techniques (e.g., FID) struggle to capture the structural properties and correlations of such datasets. In this work, we propose Struct-Bench, a framework and benchmark for evaluating synthetic datasets derived from structured datasets that contain natural language data. The Struct-Bench framework requires users to provide a representation of their dataset structure as a Context-Free Grammar (CFG). Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with CFGs. We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods. Struct-Bench also includes reference implementations of different metrics and a leaderboard, thereby providing researchers a standardized evaluation platform to benchmark and investigate privacy-preserving synthetic data generation methods. Further, we also present a case study showing how to use Struct-Bench to improve the synthetic data quality of Private Evolution (PE) on structured data. The benchmark and the leaderboard have been publicly made available at https://struct-bench.github.io.

Related papers

• A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance [8.655727081754241]
  This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets.<n>We introduce a schema eliciting key information about the goals, scope, and context of an evaluation task, and use this information to guide synthetic data generation.<n>We show how expert-informed scaffolds can guide synthetic data generation toward more valid benchmarks.
  arXiv  Detail & Related papers  (2026-09-15T03:36:35Z)
• OmniStruct: Universal Text-to-Structure Generation across Diverse Schemas [57.49565459553627]
  We introduce OmniStruct, a benchmark for assessing Large Language Models' capabilities on text-to-structure tasks.<n>We collect high-quality training data via synthetic task generation to facilitate the development of efficient text-to-structure models.<n>Our experiments demonstrate the possibility of fine-tuning much smaller models on synthetic data into universal structured generation models.
  arXiv  Detail & Related papers  (2025-11-23T08:18:12Z)
• Understanding the Influence of Synthetic Data for Text Embedders [52.04771455432998]
  We first reproduce and publicly release the synthetic data proposed by Wang et al.<n>We critically examine where exactly synthetic data improves model generalization.<n>Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders.
  arXiv  Detail & Related papers  (2025-09-07T19:28:52Z)
• StructSynth: Leveraging LLMs for Structure-Aware Tabular Data Synthesis in Low-Data Regimes [15.476662936746989]
  Struct Synth is a novel framework that integrates the generative power of Large Language Models with robust structural control.<n>It produces synthetic data with significantly higher structural integrity and downstream utility than state-of-the-art methods.<n>It proves especially effective in challenging low-data scenarios, successfully navigating the trade-off between privacy preservation and statistical fidelity.
  arXiv  Detail & Related papers  (2025-08-04T16:55:02Z)
• StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation [8.251302684712773]
  StructText is an end-to-end framework for automatically generating high-fidelity benchmarks for key-value extraction from text.<n>We evaluate the proposed method on 71,539 examples across 49 documents.
  arXiv  Detail & Related papers  (2025-07-28T21:20:44Z)
• Symbol-based entity marker highlighting for enhanced text mining in materials science with generative AI [4.178382980763478]
  We propose a novel hybrid text-mining framework to convert unstructured scientific text into structured data.<n>Our approach first transforms raw text into entity-recognized text, and subsequently into structured form.<n>We also enhance entity recognition performance by introducing an entity marker-a simple yet effective technique that uses symbolic annotations.
  arXiv  Detail & Related papers  (2025-05-09T07:58:30Z)
• Structured Evaluation of Synthetic Tabular Data [6.418460620178983]
  Tabular data is common yet typically incomplete, small in volume, and access-restricted due to privacy concerns. We propose an evaluation framework with a single, mathematical objective that posits that the synthetic data should be drawn from the same distribution as the observed data. We evaluate structurally informed synthesizers and synthesizers powered by deep learning.
  arXiv  Detail & Related papers  (2024-03-15T15:58:37Z)
• Unifying Structured Data as Graph for Data-to-Text Pre-Training [69.96195162337793]
  Data-to-text (D2T) generation aims to transform structured data into natural language text. Data-to-text pre-training has proved to be powerful in enhancing D2T generation. We propose a structure-enhanced pre-training method for D2T generation by designing a structure-enhanced Transformer.
  arXiv  Detail & Related papers  (2024-01-02T12:23:49Z)
• GEMv2: Multilingual NLG Benchmarking in a Single Line of Code [161.1761414080574]
  Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
  arXiv  Detail & Related papers  (2022-06-22T17:52:30Z)
• Learning to Synthesize Data for Semantic Parsing [57.190817162674875]
  We propose a generative model which models the composition of programs and maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. We evaluate our method in both in-domain and out-of-domain settings of text-to-Query parsing on the standard benchmarks of GeoQuery and Spider.
  arXiv  Detail & Related papers  (2021-04-12T21:24:02Z)
• A Framework for End-to-End Learning on Semantic Tree-Structured Data [4.241801379755808]
  A common form of structured data is what we term "semantic tree-structures" We propose a novel framework for end-to-end learning on generic semantic tree-structured data.
  arXiv  Detail & Related papers  (2020-02-13T18:49:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.