Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai

• URL: http://arxiv.org/abs/2411.15484v1
• Date: Sat, 23 Nov 2024 07:50:59 GMT
• Title: Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai
• Authors: Parinthapat Pengpun, Can Udomcharoenchaikit, Weerayut Buaphet, Peerat Limkonchotiwat,
• Abstract summary: We present a synthetic data approach for instruction-tuning large language models (LLMs) for low-resource languages in a data-efficient manner, specifically focusing on Thai. We identify three key properties that contribute to the effectiveness of instruction-tuning datasets: fluency, diversity, and cultural context. Our framework employs an LLM to generate diverse topics, retrieve relevant contexts from Wikipedia, and create instructions for various tasks, such as question answering, summarization, and conversation.
• Score: 5.670682861458055
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a synthetic data approach for instruction-tuning large language models (LLMs) for low-resource languages in a data-efficient manner, specifically focusing on Thai. We identify three key properties that contribute to the effectiveness of instruction-tuning datasets: fluency, diversity, and cultural context. We propose a seed-data-free framework for generating synthetic instruction-tuning data that incorporates these essential properties. Our framework employs an LLM to generate diverse topics, retrieve relevant contexts from Wikipedia, and create instructions for various tasks, such as question answering, summarization, and conversation. The experimental results show that our best-performing synthetic dataset, which incorporates all three key properties, achieves competitive performance using only 5,000 instructions when compared to state-of-the-art Thai LLMs trained on hundreds of thousands of instructions. Our code and dataset are publicly available at https://github.com/parinzee/seed-free-synthetic-instruct.

Related papers

• Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data [0.2687400480679652]
  This study proposes a pipeline for generating synthetic data and a comprehensive approach for investigating the factors that influence the validity of synthetic data generated by Large Language Models. Our results show that, in most cases and across different metrics, the fine-tuned models trained on synthetic data consistently outperformed other models on both real and synthetic test datasets.
  arXiv  Detail & Related papers  (2025-03-31T13:22:34Z)
• Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models [22.16558378953053]
  We build state-of-the-art instruction-tuning datasets sourced from human-written instructions. LLMs fine-tuned on our datasets consistently outperform those fine-tuned on existing ones. Analyses suggest that instruction-tuning in a new language allows LLMs to follow instructions, while the tuned models exhibit a notable lack of culture-specific knowledge in that language.
  arXiv  Detail & Related papers  (2025-03-31T04:28:38Z)
• ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classification [27.023332376571677]
  ARISE is a framework that iteratively induces rules and generates synthetic data for text classification. We induce rules via inductive generalisation of syntactic n-grams, enabling us to capture a complementary source of supervision.
  arXiv  Detail & Related papers  (2025-02-09T14:39:01Z)
• AIDE: Task-Specific Fine Tuning with Attribute Guided Multi-Hop Data Expansion [15.916595953695603]
  Fine-tuning large language models (LLMs) for specific tasks requires high-quality, diverse training data relevant to the task. Recent research has leveraged LLMs to synthesize training data, but existing approaches either depend on large seed datasets or struggle to ensure both task relevance and data diversity in the generated outputs. We propose AIDE, a novel data synthesis framework that uses a multi-hop process to expand 10 seed data points while ensuring diversity and task relevance.
  arXiv  Detail & Related papers  (2024-12-09T01:39:16Z)
• Efficacy of Synthetic Data as a Benchmark [3.2968976262860408]
  We investigate the effectiveness of generating synthetic data through large language models (LLMs) Our experiments show that while synthetic data can effectively capture performance of various methods for simpler tasks, it falls short for more complex tasks like named entity recognition. We propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks.
  arXiv  Detail & Related papers  (2024-09-18T13:20:23Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• EPIC: Effective Prompting for Imbalanced-Class Data Synthesis in Tabular Data Classification via Large Language Models [39.347666307218006]
  Large language models (LLMs) have demonstrated remarkable in-context learning capabilities across diverse applications. We introduce EPIC, a novel approach that leverages balanced, grouped data samples and consistent formatting with unique variable mapping to guide LLMs in generating accurate synthetic data across all classes, even for imbalanced datasets.
  arXiv  Detail & Related papers  (2024-04-15T17:49:16Z)
• CodecLM: Aligning Language Models with Tailored Synthetic Data [51.59223474427153]
  We introduce CodecLM, a framework for adaptively generating high-quality synthetic data for instruction-following abilities. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples.
  arXiv  Detail & Related papers  (2024-04-08T21:15:36Z)
• Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples [13.946626388239443]
  We aim to improve sentence embeddings without using large manually annotated datasets. We will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage few-shot examples.
  arXiv  Detail & Related papers  (2024-02-23T06:33:51Z)
• Improving Text Embeddings with Large Language Models [59.930513259982725]
  We introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data.
  arXiv  Detail & Related papers  (2023-12-31T02:13:18Z)
• TarGEN: Targeted Data Generation with Large Language Models [51.87504111286201]
  TarGEN is a multi-step prompting strategy for generating high-quality synthetic datasets. We augment TarGEN with a method known as self-correction empowering LLMs to rectify inaccurately labeled instances. A comprehensive analysis of the synthetic dataset compared to the original dataset reveals similar or higher levels of dataset complexity and diversity.
  arXiv  Detail & Related papers  (2023-10-27T03:32:17Z)
• From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models [6.520584613661788]
  We construct a Japanese instruction dataset by expanding and filtering existing datasets. We perform Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models.
  arXiv  Detail & Related papers  (2023-09-07T00:14:37Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)
• Explaining Patterns in Data with Language Models via Interpretable Autoprompting [143.4162028260874]
  We introduce interpretable autoprompting (iPrompt), an algorithm that generates a natural-language string explaining the data. iPrompt can yield meaningful insights by accurately finding groundtruth dataset descriptions. Experiments with an fMRI dataset show the potential for iPrompt to aid in scientific discovery.
  arXiv  Detail & Related papers  (2022-10-04T18:32:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.