SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT

• URL: http://arxiv.org/abs/2305.19589v1
• Date: Wed, 31 May 2023 06:22:07 GMT
• Title: SLABERT Talk Pretty One Day: Modeling Second Language Acquisition with BERT
• Authors: Aditya Yadavalli, Alekhya Yadavalli, Vera Tobin
• Abstract summary: Cross-linguistic transfer is the influence of linguistic structure of a speaker's native language on the successful acquisition of a foreign language. We find that NLP literature has not given enough attention to the phenomenon of negative transfer. Our findings call for further research using our novel Transformer-based SLA models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker's native language [L1] on the successful acquisition of a foreign language [L2]. Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition). We find that NLP literature has not given enough attention to the phenomenon of negative transfer. To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs. Further, we build a Mutlilingual Age Ordered CHILDES (MAO-CHILDES) -- a dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese -- to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2]. To examine the impact of native CDS, we use the TILT-based cross lingual transfer learning approach established by Papadimitriou and Jurafsky (2020) and find that, as in human SLA, language family distance predicts more negative transfer. Additionally, we find that conversational speech data shows greater facilitation for language acquisition than scripted speech data. Our findings call for further research using our novel Transformer-based SLA models and we would like to encourage it by releasing our code, data, and models.

Related papers

• Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education [3.799331337558008]
  Large language models (LLMs) offer promise in generating educational content, providing instructor feedback, and reducing teacher workload on assessments. We study the effectiveness of multilingual large language models (MLLMs) across monolingual (English-only, Spanish-only) and bilingual (Spanglish) student writing.
  arXiv  Detail & Related papers  (2024-11-06T23:16:25Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets [4.653113033432781]
  Cross-lingual transfer capabilities of Multilingual Language Models (MLLMs) are investigated. Our research provides valuable insights into cross-lingual transfer and its implications for NLP applications.
  arXiv  Detail & Related papers  (2024-03-29T08:47:15Z)
• Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models? [48.88328580373103]
  We propose two parameter-free $textbfL$anguage $textbfR$epresentation $textbfP$rojection modules (LRP2) The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages.
  arXiv  Detail & Related papers  (2023-11-07T08:16:16Z)
• Second Language Acquisition of Neural Language Models [17.356128991925576]
  This work sheds light on the second language (L2) acquisition of neural language models (LMs) We trained bilingual LMs with a scenario similar to human L2 acquisition and analyzed their cross-lingual transfer from linguistic perspectives.
  arXiv  Detail & Related papers  (2023-06-05T14:32:41Z)
• How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning [14.02101305717738]
  Multilingual large language models (MLLMs) are jointly trained on data from many different languages. It remains unclear to what extent, and under which conditions, languages rely on each other's data. We find that MLLMs rely on data from multiple languages from the early stages of fine-tuning and that this reliance gradually increases as fine-tuning progresses.
  arXiv  Detail & Related papers  (2023-05-22T17:47:41Z)
• Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability [16.01088313166145]
  We investigate the role of incidental bilingualism in large language models. We show that PaLM is exposed to over 30 million translation pairs across at least 44 languages. We show that its presence has a substantial impact on translation capabilities, although this impact diminishes with model scale.
  arXiv  Detail & Related papers  (2023-05-17T14:58:06Z)
• Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models [67.19567060894563]
  Pretrained Language Models (PLMs) learn rich cross-lingual knowledge and can be finetuned to perform well on diverse tasks. We present a new study investigating how well PLMs capture cross-lingual word sense with Contextual Word-Level Translation (C-WLT) We find that as the model size increases, PLMs encode more cross-lingual word sense knowledge and better use context to improve WLT performance.
  arXiv  Detail & Related papers  (2023-04-26T19:55:52Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• Exposing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders [85.80950708769923]
  We probe multilingual language models for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a novel method to expose this knowledge by additionally fine-tuning multilingual models. We report substantial gains on standard benchmarks.
  arXiv  Detail & Related papers  (2022-04-30T13:23:16Z)
• Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure [54.01613740115601]
  We study three language properties: constituent order, composition and word co-occurrence. Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.
  arXiv  Detail & Related papers  (2022-03-16T07:09:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.