Parameterized Synthetic Text Generation with SimpleStories

• URL: http://arxiv.org/abs/2504.09184v2
• Date: Fri, 16 May 2025 11:38:19 GMT
• Title: Parameterized Synthetic Text Generation with SimpleStories
• Authors: Lennart Finke, Chandan Sreedhara, Thomas Dooms, Mat Allen, Emerald Zhang, Juan Diego Rodriguez, Noa Nabeshima, Thomas Marshall, Dan Braun,
• Abstract summary: We present a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese.<n>Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale.<n>We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process.
• Score: 0.9503060186757711
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language.

Related papers

• Transfer of Structural Knowledge from Synthetic Languages [0.0]
  This work explores transfer learning from several synthetic languages to English.<n>We introduce a new synthetic language that leads to better transfer to English than the languages used in previous research.<n>We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.
  arXiv  Detail & Related papers  (2025-05-21T17:18:51Z)
• Towards Visual Text Grounding of Multimodal Large Language Model [88.0588924255417]
  We introduce TRIG, a novel task with a newly designed instruction dataset for benchmarking text-rich image grounding. Specifically, we propose an OCR-LLM-human interaction pipeline to create 800 manually annotated question-answer pairs as a benchmark. A comprehensive evaluation of various MLLMs on our proposed benchmark exposes substantial limitations in their grounding capability on text-rich images.
  arXiv  Detail & Related papers  (2025-04-07T12:01:59Z)
• BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training [1.8817715864806608]
  We study the effect of synthetic story data in language pre-training using TinyStories.<n>We train GPT-Neo models on subsets of TinyStories, while varying the amount of available data.<n>We find that, even with access to less than 100M words, the models are able to generate high-quality, original completions to a given story.
  arXiv  Detail & Related papers  (2024-10-20T11:47:17Z)
• Instruction Data Generation and Unsupervised Adaptation for Speech Language Models [21.56355461403427]
  We propose three methods for generating synthetic samples to train and evaluate multimodal large language models. Synthetic data generation emerges as a crucial strategy to enhance the performance of such systems. We highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions.
  arXiv  Detail & Related papers  (2024-06-18T08:27:00Z)
• SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation [55.2480439325792]
  We study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance.
  arXiv  Detail & Related papers  (2024-05-16T12:22:41Z)
• Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Dataset [50.09271028495819]
  multimodal research related to touch focuses on visual and tactile modalities. We construct a touch-language-vision dataset named TLV (Touch-Language-Vision) by human-machine cascade collaboration.
  arXiv  Detail & Related papers  (2024-03-14T19:01:54Z)
• German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data [0.7059555559002345]
  This study pioneers the use of synthetically generated data for training generative models in document-level text simplification of German texts. We finetune Large Language Models with up to 13 billion parameters on this data and evaluate their performance.
  arXiv  Detail & Related papers  (2024-02-16T13:28:44Z)
• Text2Data: Low-Resource Data Generation with Textual Control [100.5970757736845]
  Text2Data is a novel approach that utilizes unlabeled data to understand the underlying data distribution.<n>It undergoes finetuning via a novel constraint optimization-based learning objective that ensures controllability and effectively counteracts catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-02-08T03:41:39Z)
• Improving Text Embeddings with Large Language Models [59.930513259982725]
  We introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data.
  arXiv  Detail & Related papers  (2023-12-31T02:13:18Z)
• Split and Rephrase with Large Language Models [2.499907423888049]
  Split and Rephrase (SPRP) task consists in splitting complex sentences into a sequence of shorter grammatical sentences. We evaluate large language models on the task, showing that they can provide large improvements over the state of the art on the main metrics.
  arXiv  Detail & Related papers  (2023-12-18T10:16:37Z)
• Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training [29.143887057933327]
  We introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact.
  arXiv  Detail & Related papers  (2023-09-19T21:01:40Z)
• Unified Model Learning for Various Neural Machine Translation [63.320005222549646]
  Existing machine translation (NMT) studies mainly focus on developing dataset-specific models. We propose a versatile'' model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks. OurNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs.
  arXiv  Detail & Related papers  (2023-05-04T12:21:52Z)
• mFACE: Multilingual Summarization with Factual Consistency Evaluation [79.60172087719356]
  Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent summaries. We leverage factual consistency evaluation models to improve multilingual summarization.
  arXiv  Detail & Related papers  (2022-12-20T19:52:41Z)
• Data-to-text Generation with Variational Sequential Planning [74.3955521225497]
  We consider the task of data-to-text generation, which aims to create textual output from non-linguistic input. We propose a neural model enhanced with a planning component responsible for organizing high-level information in a coherent and meaningful way. We infer latent plans sequentially with a structured variational model, while interleaving the steps of planning and generation.
  arXiv  Detail & Related papers  (2022-02-28T13:17:59Z)
• Evaluation of Abstractive Summarisation Models with Machine Translation in Deliberative Processes [23.249742737907905]
  This dataset reflects difficulties of combining multiple narratives, mostly of poor grammatical quality, in a single text. We report an extensive evaluation of a wide range of abstractive summarisation models in combination with an off-the-shelf machine translation model. We obtain promising results regarding the fluency, consistency and relevance of the summaries produced.
  arXiv  Detail & Related papers  (2021-10-12T09:23:57Z)
• Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events [39.577220559911055]
  We propose a new task named "Outline to Story" (O2S) as a test bed for fine-grained controllable generation of long text. We then create datasets for future benchmarks, built by state-of-the-art keyword extraction techniques.
  arXiv  Detail & Related papers  (2021-01-04T08:16:21Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)
• Exemplar-Controllable Paraphrasing and Translation using Bitext [57.92051459102902]
  We adapt models from prior work to be able to learn solely from bilingual text (bitext) Our single proposed model can perform four tasks: controlled paraphrase generation in both languages and controlled machine translation in both language directions.
  arXiv  Detail & Related papers  (2020-10-12T17:02:50Z)
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance [99.68851329919858]
  discrete nature of natural language makes disentangling of textual representations more challenging. Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation.
  arXiv  Detail & Related papers  (2020-06-01T03:36:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.