Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining

• URL: http://arxiv.org/abs/2603.24826v1
• Date: Wed, 25 Mar 2026 21:29:14 GMT
• Title: Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
• Abstract summary: We study how synthetic rewriting interacts with source data quality in the context of Portuguese continued pretraining.<n>Our results demonstrate that synthetic rewriting acts primarily as a quality multiplier rather than a substitute for data curation.
• Score: 12.34636448485891
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthetic data generation through document rewriting has emerged as a promising technique for improving language model pretraining, yet most studies focus on English and do not systematically control for the quality of the source data being rewritten. We present a controlled study of how synthetic rewriting interacts with source data quality in the context of Portuguese continued pretraining. Starting from ClassiCC-PT, a Portuguese corpus annotated with STEM and Educational quality scores, we construct two 10B-token subsets at different quality levels and rewrite each into four styles using a 7B instruction-tuned model, producing approximately 40B tokens of synthetic data per condition. We train two English-centric base models (1.1B and 7B parameters) on each condition and evaluate on PoETa V2, a comprehensive 44-task Portuguese benchmark. At the 7B scale, rewriting high-quality data yields a +3.4 NPM gain over the same data unmodified, while rewriting low-quality data provides only +0.5 NPM. At the 1.1B scale, this interaction is weaker, with unmodified low-quality data performing comparably to rewritten high-quality data. Our results demonstrate that synthetic rewriting acts primarily as a quality multiplier rather than a substitute for data curation, and that this effect is scale-dependent.

Related papers

• Hierarchical Policy Optimization for Simultaneous Translation of Unbounded Speech [50.45710815530982]
  Simultaneous speech translation (SST) generates translations while receiving partial speech input.<n>Recent advances show that large language models (LLMs) can substantially improve SST quality, but at the cost of high computational overhead.<n>We propose a Hierarchical Policy Optimization (HPO) approach that trains post-train models on imperfect SFT data.<n> Experiments on English to Chinese/German/Japanese demonstrate improvements of over +7 COMET score and +1.25 MetricX score at a latency of 1.5 seconds.
  arXiv  Detail & Related papers  (2026-04-22T19:43:51Z)
• ENTP: Enhancing Low-Quality SFT Data via Neural-Symbolic Text Purge-Mix [31.617919238979013]
  Supervised Fine-Tuning (SFT) adapts pre-trained Large Language Models (LLMs) to domain-specific instructions by training on a subset of high-quality instruction-response pairs.<n>Existing quality-first paradigms often overlook valuable signals in discarded low-quality data and rely on imperfect quality filters.<n>We introduce ENTP (Enhancing low-quality SFT data via Neural-symbolic Text Purge-Mix), a framework that revitalizes low-quality corpora through symbolic purification and neural reconstruction.
  arXiv  Detail & Related papers  (2025-10-27T09:39:22Z)
• Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation [55.73341401764367]
  We introduce DCSQE, a novel framework for alleviating distribution shift in synthetic QE data.<n> DCSQE uses references, i.e., translation supervision signals, to guide both the generation and annotation processes.<n>Experiments demonstrate that DCSQE outperforms SOTA baselines in both supervised and unsupervised settings.
  arXiv  Detail & Related papers  (2025-02-27T10:11:53Z)
• DataMan: Data Manager for Pre-training Large Language Models [39.677609311769146]
  Existing methods rely on limited intuition, lacking comprehensive and clear guidelines.<n>We derive 14 quality criteria from the causes of text perplexity anomalies and introduce 15 common application domains to support domain mixing.<n>Our experiments validate our approach, using DataMan to select 30B tokens to train a 1.3B- parameter language model.
  arXiv  Detail & Related papers  (2025-02-26T18:01:19Z)
• Evaluating Language Models as Synthetic Data Generators [99.16334775127875]
  AgoraBench is a benchmark that provides standardized settings and metrics to evaluate LMs' data generation abilities.<n>Through synthesizing 1.26 million training instances using 6 LMs and training 99 student models, we uncover key insights about LMs' data generation capabilities.
  arXiv  Detail & Related papers  (2024-12-04T19:20:32Z)
• Scaling Parameter-Constrained Language Models with Quality Data [32.35610029333478]
  Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters. We extend the conventional understanding of scaling law by offering a microscopic view of data quality within the original formulation.
  arXiv  Detail & Related papers  (2024-10-04T02:07:17Z)
• Text Quality-Based Pruning for Efficient Training of Language Models [66.66259229732121]
  We propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets. By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances. Experimental results over multiple models and datasets demonstrate the efficacy of this approach.
  arXiv  Detail & Related papers  (2024-04-26T18:01:25Z)
• QuRating: Selecting High-Quality Data for Training Language Models [64.83332850645074]
  We introduce QuRating, a method for selecting pre-training data that can capture human intuitions about data quality. In this paper, we investigate four qualities - writing style, required expertise, facts & trivia, and educational value. We train a Qur model to learn scalar ratings from pairwise judgments, and use it to annotate a 260B training corpus with quality ratings for each of the four criteria.
  arXiv  Detail & Related papers  (2024-02-15T06:36:07Z)
• NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality [123.97136358092585]
  We develop a TTS system called NaturalSpeech that achieves human-level quality on a benchmark dataset. Specifically, we leverage a variational autoencoder (VAE) for end-to-end text to waveform generation. Experiment evaluations on popular LJSpeech dataset show that our proposed NaturalSpeech achieves -0.01 CMOS to human recordings at the sentence level.
  arXiv  Detail & Related papers  (2022-05-09T16:57:35Z)
• Structure-Tags Improve Text Classification for Scholarly Document Quality Prediction [4.4641025448898475]
  We propose the use of HANs combined with structure-tags which mark the role of sentences in the document. Adding tags to sentences, marking them as corresponding to title, abstract or main body text, yields improvements over the state-of-the-art for scholarly document quality prediction.
  arXiv  Detail & Related papers  (2020-04-30T22:34:34Z)
• AR: Auto-Repair the Synthetic Data for Neural Machine Translation [34.36472405208541]
  We propose a novel Auto- Repair (AR) framework to improve the quality of synthetic data. Our proposed AR model can learn the transformation from low quality (noisy) input sentence to high quality sentence. Our approach can effective improve the quality of synthetic parallel data and the NMT model with the repaired synthetic data achieves consistent improvements.
  arXiv  Detail & Related papers  (2020-04-05T13:18:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.