Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models

• URL: http://arxiv.org/abs/2404.17010v1
• Date: Thu, 25 Apr 2024 20:10:14 GMT
• Title: Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models
• Authors: Eren Dogan, M. Egemen Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, Ezgi Sevi, H. Toprak Kesgin, M. Kaan Yuce, M. Fatih Amasyali,
• Abstract summary: A comparison is made among seven selected language models based on their contextual learning and question-answering abilities. The results show that for question-answering, continuing pretraining before fine-tuning with instructional datasets is more successful in adapting multilingual models to Turkish.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The developments that language models have provided in fulfilling almost all kinds of tasks have attracted the attention of not only researchers but also the society and have enabled them to become products. There are commercially successful language models available. However, users may prefer open-source language models due to cost, data privacy, or regulations. Yet, despite the increasing number of these models, there is no comprehensive comparison of their performance for Turkish. This study aims to fill this gap in the literature. A comparison is made among seven selected language models based on their contextual learning and question-answering abilities. Turkish datasets for contextual learning and question-answering were prepared, and both automatic and human evaluations were conducted. The results show that for question-answering, continuing pretraining before fine-tuning with instructional datasets is more successful in adapting multilingual models to Turkish and that in-context learning performances do not much related to question-answering performances.

Related papers

• mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval [61.17793165194077]
  We introduce mFollowIR, a benchmark for measuring instruction-following ability in retrieval models. We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance. We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting.
  arXiv  Detail & Related papers  (2025-01-31T16:24:46Z)
• Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training [0.0]
  We adapt Large Language Model generated datasets and translated English datasets into Turkish. This approach led to substantial enhancements in model accuracy for both few-shot and zero-shot learning scenarios.
  arXiv  Detail & Related papers  (2024-12-03T19:17:18Z)
• DevBench: A multimodal developmental benchmark for language learning [0.34129029452670606]
  We introduce DevBench, a benchmark for evaluating vision-language models on tasks and behavioral data. We show that DevBench provides a benchmark for comparing models to human language development. These comparisons highlight ways in which model and human language learning processes diverge.
  arXiv  Detail & Related papers  (2024-06-14T17:49:41Z)
• Introducing cosmosGPT: Monolingual Training for Turkish Language Models [0.0]
  This study introduces the cosmosGPT models that we created with this alternative method. We then introduce new finetune datasets for basic language models to fulfill user requests and new evaluation datasets for measuring the capabilities of Turkish language models. The results show that the language models we built with the monolingual corpus have promising performance despite being about 10 times smaller than the others.
  arXiv  Detail & Related papers  (2024-04-26T11:34:11Z)
• Evaluating Large Language Models on Controlled Generation Tasks [92.64781370921486]
  We present an extensive analysis of various benchmarks including a sentence planning benchmark with different granularities. After comparing large language models against state-of-the-start finetuned smaller models, we present a spectrum showing large language models falling behind, are comparable, or exceed the ability of smaller models.
  arXiv  Detail & Related papers  (2023-10-23T03:48:24Z)
• Lessons learned from the evaluation of Spanish Language Models [27.653133576469276]
  We present a head-to-head comparison of language models for Spanish with the following results. We argue for the need of more research to understand the factors underlying them. The recent activity in the development of language technology for Spanish is to be welcomed, but our results show that building language models remains an open, resource-heavy problem.
  arXiv  Detail & Related papers  (2022-12-16T10:33:38Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Specializing Multilingual Language Models: An Empirical Study [50.7526245872855]
  Contextualized word representations from pretrained multilingual language models have become the de facto standard for addressing natural language tasks. For languages rarely or never seen by these models, directly using such models often results in suboptimal representation or use of data.
  arXiv  Detail & Related papers  (2021-06-16T18:13:55Z)
• Improving Cross-Lingual Reading Comprehension with Self-Training [62.73937175625953]
  Current state-of-the-art models even surpass human performance on several benchmarks. Previous works have revealed the abilities of pre-trained multilingual models for zero-shot cross-lingual reading comprehension. This paper further utilized unlabeled data to improve the performance.
  arXiv  Detail & Related papers  (2021-05-08T08:04:30Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• An Empirical Study of Factors Affecting Language-Independent Models [11.976665726887733]
  We show that language-independent models can be comparable to or even outperforms the models trained using monolingual data. We experiment language-independent models with many different languages and show that they are more suitable for typologically similar languages.
  arXiv  Detail & Related papers  (2019-12-30T22:41:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.