Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets

• URL: http://arxiv.org/abs/2501.15624v1
• Date: Sun, 26 Jan 2025 18:10:20 GMT
• Title: Improving Estonian Text Simplification through Pretrained Language Models and Custom Datasets
• Authors: Eduard Barbu, Meeri-Ly Muru, Sten Marcus Malva,
• Abstract summary: This study introduces an approach to Estonian text simplification using a neural machine translation model and a fine-tuned large language model (LLaMA) We benchmarked OpenNMT, a neural machine translation model that frames text simplification as a translation task, and fine-tuned the LLaMA model on our dataset to tailor it specifically for Estonian simplification. Manual evaluations on the test set show that the LLaMA model consistently outperforms OpenNMT in readability, grammaticality, and meaning preservation.
• Score: 0.0
• License:
• Abstract: This study introduces an approach to Estonian text simplification using two model architectures: a neural machine translation model and a fine-tuned large language model (LLaMA). Given the limited resources for Estonian, we developed a new dataset, the Estonian Simplification Dataset, combining translated data and GPT-4.0-generated simplifications. We benchmarked OpenNMT, a neural machine translation model that frames text simplification as a translation task, and fine-tuned the LLaMA model on our dataset to tailor it specifically for Estonian simplification. Manual evaluations on the test set show that the LLaMA model consistently outperforms OpenNMT in readability, grammaticality, and meaning preservation. These findings underscore the potential of large language models for low-resource languages and provide a basis for further research in Estonian text simplification.

Related papers

• GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian [0.21485350418225246]
  We present GliLem, a novel hybrid lemmatization system for Estonian. We leverage the flexibility of a pre-trained GliNER model to improve the lemmatization accuracy of Vabamorf.
  arXiv  Detail & Related papers  (2024-12-29T22:02:00Z)
• Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer [0.7499722271664147]
  We investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining on Estonian. We showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning. Our best model, named textscLlammas, represents the first open-source instruction-following LLM for Estonian.
  arXiv  Detail & Related papers  (2024-04-05T11:52:02Z)
• German Text Simplification: Finetuning Large Language Models with Semi-Synthetic Data [0.7059555559002345]
  This study pioneers the use of synthetically generated data for training generative models in document-level text simplification of German texts. We finetune Large Language Models with up to 13 billion parameters on this data and evaluate their performance.
  arXiv  Detail & Related papers  (2024-02-16T13:28:44Z)
• Unsupervised Lexical Simplification with Context Augmentation [55.318201742039]
  Given a target word and its context, our method generates substitutes based on the target context and additional contexts sampled from monolingual data. We conduct experiments in English, Portuguese, and Spanish on the TSAR-2022 shared task, and show that our model substantially outperforms other unsupervised systems across all languages.
  arXiv  Detail & Related papers  (2023-11-01T05:48:05Z)
• Unified Model Learning for Various Neural Machine Translation [63.320005222549646]
  Existing machine translation (NMT) studies mainly focus on developing dataset-specific models. We propose a versatile'' model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks. OurNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs.
  arXiv  Detail & Related papers  (2023-05-04T12:21:52Z)
• Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation [50.54059385277964]
  We present a novel data augmentation paradigm termed Continuous Semantic Augmentation (CsaNMT) CsaNMT augments each training instance with an adjacency region that could cover adequate variants of literal expression under the same meaning.
  arXiv  Detail & Related papers  (2022-04-14T08:16:28Z)
• TunBERT: Pretrained Contextualized Text Representation for Tunisian Dialect [0.0]
  We investigate the feasibility of training monolingual Transformer-based language models for under represented languages. We show that the use of noisy web crawled data instead of structured data is more convenient for such non-standardized language. Our best performing TunBERT model reaches or improves the state-of-the-art in all three downstream tasks.
  arXiv  Detail & Related papers  (2021-11-25T15:49:50Z)
• Factorized Neural Transducer for Efficient Language Model Adaptation [51.81097243306204]
  We propose a novel model, factorized neural Transducer, by factorizing the blank and vocabulary prediction. It is expected that this factorization can transfer the improvement of the standalone language model to the Transducer for speech recognition. We demonstrate that the proposed factorized neural Transducer yields 15% to 20% WER improvements when out-of-domain text data is used for language model adaptation.
  arXiv  Detail & Related papers  (2021-09-27T15:04:00Z)
• Improving Neural Machine Translation by Bidirectional Training [85.64797317290349]
  We present a simple and effective pretraining strategy -- bidirectional training (BiT) for neural machine translation. Specifically, we bidirectionally update the model parameters at the early stage and then tune the model normally. Experimental results show that BiT pushes the SOTA neural machine translation performance across 15 translation tasks on 8 language pairs significantly higher.
  arXiv  Detail & Related papers  (2021-09-16T07:58:33Z)
• Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation [59.191079800436114]
  Document-level machine translation incorporates inter-sentential dependencies into the translation of a source sentence. We propose a new framework to model cross-sentence dependencies by training neural machine translation (NMT) to predict both the target translation and surrounding sentences of a source sentence.
  arXiv  Detail & Related papers  (2020-03-30T03:38:01Z)
• Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling [4.525267347429154]
  We train a Transformer-based neural model on the BERT language model. In addition, we propose a new method of BERT-windowing, which allows chunk-wise processing of texts longer than the BERT window size. The results of our models are compared to a baseline and the state-of-the-art models on the CNN/Daily Mail dataset.
  arXiv  Detail & Related papers  (2020-03-29T14:00:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.