BAMBI: Developing Baby Language Models for Italian

• URL: http://arxiv.org/abs/2503.09481v1
• Date: Wed, 12 Mar 2025 15:36:50 GMT
• Title: BAMBI: Developing Baby Language Models for Italian
• Authors: Alice Suozzi, Luca Capone, Gianluca E. Lebani, Alessandro Lenci,
• Abstract summary: This paper presents BAMBI (BAby language Models Boostrapped for Italian), a series of Baby Language Models (BabyLMs) trained on data that mimic the linguistic input received by a five-year-old Italian-speaking child.<n>The BAMBI models are tested using a benchmark specifically designed to evaluate language models, which takes into account the amount of training input the models received.
• Score: 45.36413940519089
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper presents BAMBI (BAby language Models Boostrapped for Italian), a series of Baby Language Models (BabyLMs) trained on data that mimic the linguistic input received by a five-year-old Italian-speaking child. The BAMBI models are tested using a benchmark specifically designed to evaluate language models, which takes into account the amount of training input the models received. The BAMBI models are compared against a large language model (LLM) and a multimodal language model (VLM) to study the contribution of extralinguistic information for language acquisition. The results of our evaluation align with the existing literature on English language models, confirming that while reduced training data support the development of relatively robust syntactic competence, they are insufficient for fostering semantic understanding. However, the gap between the training resources (data and computation) of the BAMBI models and the LLMs is not fully reflected in their performance: despite LLMs' massive training, their performance is not much better than that of BAMBI models. This suggests that strategies beyond scaling training resources, such as data curation, inclusion of multimodal input, and other training strategies such as curriculum learning, could play a crucial role in shaping model performance.

Related papers

• LLMic: Romanian Foundation Language Model [76.09455151754062]
  We present LLMic, a foundation language model designed specifically for the Romanian Language.<n>We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks.
  arXiv  Detail & Related papers  (2025-01-13T22:14:45Z)
• Is Child-Directed Speech Effective Training Data for Language Models? [34.46268640655943]
  We train GPT-2 and RoBERTa models on 29M words of English child-directed speech. We test whether the global developmental ordering or the local discourse ordering of children's training data supports high performance relative to other datasets. These findings support the hypothesis that, rather than proceeding from better data, the child's learning algorithm is substantially more data-efficient than current language modeling techniques.
  arXiv  Detail & Related papers  (2024-08-07T08:18:51Z)
• MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting [53.77590764277568]
  We introduce a novel MoE-CT architecture that separates the base model's learning from the multilingual expansion process. Our design freezes the original LLM parameters, thus safeguarding its performance in high-resource languages, while an appended MoE module, trained on diverse language datasets, augments low-resource language proficiency.
  arXiv  Detail & Related papers  (2024-06-25T11:03:45Z)
• BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM [3.329407751651262]
  We introduce BAMBINO-LM, a continual pre-training strategy for small-scale language models. We show that BAMBINO-LM improves the Italian language capability of a BabyLM baseline. We also show that, as a side effect, the proposed method leads to a similar degradation in L1 effectiveness as human children would have had in an equivalent learning scenario.
  arXiv  Detail & Related papers  (2024-06-17T11:08:08Z)
• DevBench: A multimodal developmental benchmark for language learning [0.34129029452670606]
  We introduce DevBench, a benchmark for evaluating vision-language models on tasks and behavioral data. We show that DevBench provides a benchmark for comparing models to human language development. These comparisons highlight ways in which model and human language learning processes diverge.
  arXiv  Detail & Related papers  (2024-06-14T17:49:41Z)
• The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights [108.40766216456413]
  We propose a question alignment framework to bridge the gap between large language models' English and non-English performance. Experiment results show it can boost multilingual performance across diverse reasoning scenarios, model families, and sizes. We analyze representation space, generated response and data scales, and reveal how question translation training strengthens language alignment within LLMs.
  arXiv  Detail & Related papers  (2024-05-02T14:49:50Z)
• Tele-FLM Technical Report [96.19923831660266]
  We introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model. It features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. It is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B.
  arXiv  Detail & Related papers  (2024-04-25T14:34:47Z)
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition [4.779196219827507]
  We argue that some of the most prominent benchmarks for evaluating the syntactic capacities of neural language models may not be sufficiently rigorous. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.
  arXiv  Detail & Related papers  (2023-10-31T00:16:17Z)
• Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages [70.25418443146435]
  We propose an efficient modeling framework for cross-lingual named entity recognition in semi-structured text data. We employ two independent datasets of SMSs in English and Arabic, each carrying semi-structured banking transaction information. With access to only 30 labeled samples, our model can generalize the recognition of merchants, amounts, and other fields from English to Arabic.
  arXiv  Detail & Related papers  (2023-07-16T00:45:42Z)
• EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning [38.928786416891924]
  We introduce efficient and effective massively multilingual sentence embedding (EMS) using cross-lingual token-level reconstruction (XTR) and sentence-level contrastive learning as training objectives. Compared with related studies, the proposed model can be efficiently trained using significantly fewer parallel sentences and GPU computation resources. We release the codes for model training and the EMS pre-trained sentence embedding model, which supports 62 languages.
  arXiv  Detail & Related papers  (2022-05-31T12:29:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.