Native Language Identification with Large Language Models

• URL: http://arxiv.org/abs/2312.07819v1
• Date: Wed, 13 Dec 2023 00:52:15 GMT
• Title: Native Language Identification with Large Language Models
• Authors: Wei Zhang and Alexandre Salle
• Abstract summary: We show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes.
• Score: 60.80452362519818
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the first experiments on Native Language Identification (NLI) using LLMs such as GPT-4. NLI is the task of predicting a writer's first language by analyzing their writings in a second language, and is used in second language acquisition and forensic linguistics. Our results show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark TOEFL11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes, which has practical implications for real-world applications. Finally, we also show that LLMs can provide justification for their choices, providing reasoning based on spelling errors, syntactic patterns, and usage of directly translated linguistic patterns.

Related papers

• Leveraging Open-Source Large Language Models for Native Language Identification [1.6267479602370543]
  Native Language Identification (NLI) has applications in forensics, marketing, and second language acquisition. This study explores the potential of using open-source generative large language models (LLMs) for NLI.
  arXiv  Detail & Related papers  (2024-09-15T08:14:18Z)
• Exploring Design Choices for Building Language-Specific LLMs [36.32622880071991]
  We study building language-specific language models by adapting monolingual and multilingual models. We find that the initial performance of LLM does not always correlate with the final performance after the adaptation.
  arXiv  Detail & Related papers  (2024-06-20T18:47:43Z)
• Prompting open-source and commercial language models for grammatical error correction of English learner text [19.192210777082053]
  Large language models (LLMs) can be prompt to produce texts which are fluent and grammatical. We evaluate how well LLMs can perform at grammatical error correction (GEC) by measuring their performance on established benchmark datasets. We find that several open-source models outperform commercial ones on minimal edit benchmarks, and that in some settings zero-shot prompting is just as competitive as few-shot prompting.
  arXiv  Detail & Related papers  (2024-01-15T14:19:47Z)
• Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models [2.7013338932521416]
  We advocate for the revival of vocabulary tests as a valuable tool for assessing Large Language Models (LLMs) performance. We evaluate seven LLMs using two vocabulary test formats across two languages and uncover surprising gaps in their lexical knowledge.
  arXiv  Detail & Related papers  (2023-10-23T08:45:12Z)
• On Bilingual Lexicon Induction with Large Language Models [81.6546357879259]
  We examine the potential of the latest generation of Large Language Models for the development of bilingual lexicons. We study 1) zero-shot prompting for unsupervised BLI and 2) few-shot in-context prompting with a set of seed translation pairs. Our work is the first to demonstrate strong BLI capabilities of text-to-text mLLMs.
  arXiv  Detail & Related papers  (2023-10-21T12:43:27Z)
• Improving Domain-Specific Retrieval by NLI Fine-Tuning [64.79760042717822]
  This article investigates the fine-tuning potential of natural language inference (NLI) data to improve information retrieval and ranking. We employ both monolingual and multilingual sentence encoders fine-tuned by a supervised method utilizing contrastive loss and NLI data. Our results point to the fact that NLI fine-tuning increases the performance of the models in both tasks and both languages, with the potential to improve mono- and multilingual models.
  arXiv  Detail & Related papers  (2023-08-06T12:40:58Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• LERT: A Linguistically-motivated Pre-trained Language Model [67.65651497173998]
  We propose LERT, a pre-trained language model that is trained on three types of linguistic features along with the original pre-training task. We carried out extensive experiments on ten Chinese NLU tasks, and the experimental results show that LERT could bring significant improvements.
  arXiv  Detail & Related papers  (2022-11-10T05:09:16Z)
• FOLIO: Natural Language Reasoning with First-Order Logic [147.50480350846726]
  We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL) FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models.
  arXiv  Detail & Related papers  (2022-09-02T06:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.