Efficient Embedding-based Synthetic Data Generation for Complex Reasoning Tasks

• URL: http://arxiv.org/abs/2603.22294v1
• Date: Sun, 15 Mar 2026 00:19:03 GMT
• Title: Efficient Embedding-based Synthetic Data Generation for Complex Reasoning Tasks
• Abstract summary: We analyze the diversity and distribution of generated data in the embedding space.<n>We present a targeted pipeline for embedding-based sampling that enhances data diversity and consistently improves performance.
• Score: 8.173132368801783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthetic Data Generation (SDG), leveraging Large Language Models (LLMs), has recently been recognized and broadly adopted as an effective approach to improve the performance of smaller but more resource and compute efficient LLMs through fine-tuning. A key challenge in SDG is ensuring the quality and diversity of the generated data. In this paper, we analyze the diversity and distribution of generated data in the embedding space, and demonstrate a strong correlation between the density of examples within a specific neighborhood and the accuracy of predictions on examples drawn from that region. Building on this insight, we present a targeted pipeline for embedding-based sampling that enhances data diversity and consistently improves performance across several benchmarks.

Related papers

• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• Less is Enough: Synthesizing Diverse Data in Feature Space of LLMs [39.84010804274527]
  We introduce Feature Activation Coverage (FAC) which measures data diversity in an interpretable feature space.<n>Building upon this metric, we propose a diversity-driven data synthesis framework, named FAC Synthesis.<n>Experiments show that our approach consistently improves both data diversity and downstream performance on various tasks.
  arXiv  Detail & Related papers  (2026-02-11T00:23:13Z)
• Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMs [7.355858495660162]
  We introduce Diversity-Aware Tabular data gEnerator (DATE), a framework that prepares high-quality and distributionally distinct examples for in-context learning.<n>DATE harnesses Large Language Models (LLMs) to explore the diversity of the partitioned distribution with decision tree reasoning as feedback, generating high-quality labeled data for each subset.<n>On average, DATE achieves a 23.75% reduction in error rate with just 100 generated data.
  arXiv  Detail & Related papers  (2025-12-26T08:02:51Z)
• Curvature Enhanced Data Augmentation for Regression [4.910937238451485]
  We introduce the Curvature-Enhanced Manifold Sampling (CEMS) method for regression tasks.<n>CEMS delivers superior performance in both in-distribution and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-06-07T16:18:37Z)
• Diversity-oriented Data Augmentation with Large Language Models [9.548912625579947]
  We propose a textbfunderlineDi-textbfunderlineoriented data textbfunderlineAugmentation framework (textbfDoAug)<n>Specifically, we utilize a diversity-oriented fine-tuning approach to train an LLM as a diverse paraphraser, which is capable of augmenting textual datasets by generating diversified paraphrases.<n>The results show that our fine-tuned LLM augmenter improves diversity while preserving label consistency, thereby enhancing the robustness and performance of downstream tasks.
  arXiv  Detail & Related papers  (2025-02-17T11:00:40Z)
• Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data [54.3895971080712]
  Fine-tuning large language models (LLMs) using diverse datasets is crucial for enhancing their overall performance across various domains.<n>We propose a new method that gives the LLM a dual identity: an output model to cognitively probe and select data based on diversity reward, as well as an input model to be tuned with the selected data.
  arXiv  Detail & Related papers  (2025-02-05T17:21:01Z)
• Generating Realistic Tabular Data with Large Language Models [49.03536886067729]
  Large language models (LLM) have been used for diverse tasks, but do not capture the correct correlation between the features and the target variable. We propose a LLM-based method with three important improvements to correctly capture the ground-truth feature-class correlation in the real data. Our experiments show that our method significantly outperforms 10 SOTA baselines on 20 datasets in downstream tasks.
  arXiv  Detail & Related papers  (2024-10-29T04:14:32Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding [67.61357003974153]
  We propose a novel data augmentation framework dubbed CoDA. CoDA synthesizes diverse and informative augmented examples by integrating multiple transformations organically. A contrastive regularization objective is introduced to capture the global relationship among all the data samples.
  arXiv  Detail & Related papers  (2020-10-16T23:57:03Z)
• Improving Generative Adversarial Networks with Local Coordinate Coding [150.24880482480455]
  Generative adversarial networks (GANs) have shown remarkable success in generating realistic data from some predefined prior distribution. In practice, semantic information might be represented by some latent distribution learned from data. We propose an LCCGAN model with local coordinate coding (LCC) to improve the performance of generating data.
  arXiv  Detail & Related papers  (2020-07-28T09:17:50Z)
• Generative Data Augmentation for Commonsense Reasoning [75.26876609249197]
  G-DAUGC is a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. G-DAUGC consistently outperforms existing data augmentation methods based on back-translation. Our analysis demonstrates that G-DAUGC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.
  arXiv  Detail & Related papers  (2020-04-24T06:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.