Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models

• URL: http://arxiv.org/abs/2307.14134v1
• Date: Wed, 26 Jul 2023 12:02:30 GMT
• Title: Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models
• Authors: Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
• Abstract summary: This study introduces and evaluates tiny, mini, small, and medium-sized uncased Turkish BERT models. We trained these models on a diverse dataset encompassing over 75GB of text from multiple sources and tested them on several tasks, including mask prediction, sentiment analysis, news classification, and, zero-shot classification.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study introduces and evaluates tiny, mini, small, and medium-sized uncased Turkish BERT models, aiming to bridge the research gap in less-resourced languages. We trained these models on a diverse dataset encompassing over 75GB of text from multiple sources and tested them on several tasks, including mask prediction, sentiment analysis, news classification, and, zero-shot classification. Despite their smaller size, our models exhibited robust performance, including zero-shot task, while ensuring computational efficiency and faster execution times. Our findings provide valuable insights into the development and application of smaller language models, especially in the context of the Turkish language.

Related papers

• Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages [0.43498389175652036]
  This study integrates traditional and novel language models with fine-tuned Whisper models to raise their performance in less commonly studied languages. We demonstrate substantial improvements in word error rate, particularly in low-resource scenarios. While the integration reliably benefits all model sizes, the extent of improvement varies, highlighting the importance of optimized language model parameters.
  arXiv  Detail & Related papers  (2025-03-30T18:03:52Z)
• Scaling BERT Models for Turkish Automatic Punctuation and Capitalization Correction [0.0]
  The study presents a systematic comparison of the performance metrics precision, recall, and F1 score of each model. The results demonstrate a significant improvement in text readability and accuracy as model size increases.
  arXiv  Detail & Related papers  (2024-12-03T18:59:51Z)
• A Survey of Small Language Models [104.80308007044634]
  Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources. We present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques.
  arXiv  Detail & Related papers  (2024-10-25T23:52:28Z)
• Multilingual Models for Check-Worthy Social Media Posts Detection [0.552480439325792]
  The study includes a comprehensive analysis of different models, with a special focus on multilingual models. The novelty of this work lies in the development of multi-label multilingual classification models that can simultaneously detect harmful posts and posts that contain verifiable factual claims in an efficient way.
  arXiv  Detail & Related papers  (2024-08-13T08:55:28Z)
• Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification [4.4467858321751015]
  We benchmark language models from 77M to 40B parameters using different architectures and scoring functions. Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts. This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.
  arXiv  Detail & Related papers  (2024-04-17T07:10:28Z)
• Evaluating Large Language Models on Controlled Generation Tasks [92.64781370921486]
  We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models.
  arXiv  Detail & Related papers  (2023-10-23T03:48:24Z)
• Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages [70.25418443146435]
  We propose an efficient modeling framework for cross-lingual named entity recognition in semi-structured text data. We employ two independent datasets of SMSs in English and Arabic, each carrying semi-structured banking transaction information. With access to only 30 labeled samples, our model can generalize the recognition of merchants, amounts, and other fields from English to Arabic.
  arXiv  Detail & Related papers  (2023-07-16T00:45:42Z)
• Comparison of Pre-trained Language Models for Turkish Address Parsing [0.0]
  We focus on Turkish maps data and thoroughly evaluate both multilingual and Turkish based BERT, DistilBERT, ELECTRA and RoBERTa. We also propose a MultiLayer Perceptron (MLP) for fine-tuning BERT in addition to the standard approach of one-layer fine-tuning.
  arXiv  Detail & Related papers  (2023-06-24T12:09:43Z)
• A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models [87.7086269902562]
  We show that subword-based models might still be the most practical choice in many settings. We encourage future work in tokenizer-free methods to consider these factors when designing and evaluating new models.
  arXiv  Detail & Related papers  (2022-10-13T15:47:09Z)
• Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese [4.4681678689625715]
  We analyse the effect of pre-training with monolingual data for a low-resource language. We present a newly created corpus for Maltese, and determine the effect that the pre-training data size and domain have on the downstream performance. We compare two models on the new corpus: a monolingual BERT model trained from scratch (BERTu), and a further pre-trained multilingual BERT (mBERTu)
  arXiv  Detail & Related papers  (2022-05-21T06:44:59Z)
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model [58.27176041092891]
  Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements. We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
  arXiv  Detail & Related papers  (2020-11-23T16:00:42Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Load What You Need: Smaller Versions of Multilingual BERT [0.7734726150561088]
  We present an evaluation of smaller versions of multilingual BERT on the XNLI data set. We can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters.
  arXiv  Detail & Related papers  (2020-10-12T11:29:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.