Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM

• URL: http://arxiv.org/abs/2609.17435v1
• Date: Tue, 15 Sep 2026 16:43:32 GMT
• Title: Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
• Abstract summary: We submit MéTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track.<n>It scores 85.97 +/- 0.17% on QFrBLiMP and 62.80% on the BabyLM-weighted leaderboard.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We submit MéTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of grammatical minimal pairs) and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE (General Language Understanding Evaluation) protocol that combines French task-data translation with rank-16 LoRA (Low-Rank Adaptation) produces a sharp task-type gradient: relational tasks gain measurably, while world-knowledge tasks regress. Bilingual Lexicon Induction aligns the French embeddings to GPT-2 at p@1 = 68.84 +/- 8.61%, 18X above chance, suggesting cross-lingual alignment tracks acquired grammatical competence rather than training duration. An ablation study shows that single-token zero-shot scoring is dominated by tokenizer and template artifacts at the child scale, motivating tokenizer-swap sensitivity, placebo-controlled prompting, and native-language minimal-pair benchmarks as standard diagnostics.

Related papers

• EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languages [2.897376329231807]
  Machine translation can distort task-critical constraints and required outputs.<n>We introduce EuroAlpaca, a task-preserving localisation pipeline.<n>We show that preserving task semantics is essential for multilingual instruction tuning.
  arXiv  Detail & Related papers  (2026-09-04T12:05:53Z)
• Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya [6.396476220617138]
  We introduce a vocabulary-extended variant of XLM-R targeting the two highest-resource Ge'ez-script languages, Amharic and Tigrinya.<n>VexMLM achieves 87.0 EM /90.0 F1, versus 66.0 EM/78.0 F1 for XLM-R and 74.0 EM/ 78.0 F1 for Glot500.<n>On SA, VEXMLM reaches 80.0% accuracy versus 77.0% (XLM-R) and 46.0% (Glot500)
  arXiv  Detail & Related papers  (2026-07-16T17:08:28Z)
• Spoken Language Identification with Pre-trained Models and Margin Loss [9.800066159312749]
  For the speaker-controlled spoken language identification task proposed in the TidyLang Challenge 2026, this paper proposes a language identification method based on pre-trained models and margin-based losses.<n>The proposed method adopts a pre-trained ECAPA-TDNN as the feature encoder and incorporates margin-based losses to enhance the discriminative ability of language representations.
  arXiv  Detail & Related papers  (2026-05-03T14:37:52Z)
• Translation as a Scalable Proxy for Multilingual Evaluation [41.29816736489213]
  We evaluate the validity of a simpler alternative: can translation quality alone indicate a model's broader multilingual capabilities?<n>We find that translation performance is a good indicator of downstream task success.
  arXiv  Detail & Related papers  (2026-01-16T21:01:40Z)
• XplaiNLP at CheckThat! 2025: Multilingual Subjectivity Detection with Finetuned Transformers and Prompt-Based Inference with Large Language Models [2.749729059235755]
  This notebook reports the Xplai submission to the CheckThat! 2025 shared task on multilingual subjectivity detection.<n>We evaluate two approaches: supervised fine-tuning of transformer encoders, EuroBERT, XLM-RoBERTa, and German-BERT, on monolingual and machine-translated training data.<n>For German, a German-BERT model fine-tuned on translated training data from typologically related languages yields competitive performance over the baseline.
  arXiv  Detail & Related papers  (2025-09-15T16:53:41Z)
• SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Work [87.9341538630949]
  The first Sign Language Production Challenge was held as part of the third SLRTP Workshop at CVPR 2025.<n>The competition's aims are to evaluate architectures that translate from spoken language sentences to a sequence of skeleton poses.<n>This paper presents the challenge design and the winning methodologies.
  arXiv  Detail & Related papers  (2025-08-09T11:57:33Z)
• SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment [78.4550589538805]
  We propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism.<n> Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs.
  arXiv  Detail & Related papers  (2025-01-07T10:29:43Z)
• Extrapolating Large Language Models to Non-English by Aligning Languages [109.09051737966178]
  Existing large language models show disparate capability across different languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages.
  arXiv  Detail & Related papers  (2023-08-09T13:32:06Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE [203.65227947509933]
  This report describes our JDExplore d-team's Vega v2 submission on the SuperGLUE leaderboard. SuperGLUE is more challenging than the widely used general language understanding evaluation (GLUE) benchmark, containing eight difficult language understanding tasks.
  arXiv  Detail & Related papers  (2022-12-04T15:36:18Z)
• CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation [113.99145386490639]
  Cross-lingual NER can transfer knowledge between languages via aligned cross-lingual representations or machine translation results. We propose a Cross-lingual Entity Projection framework (CROP) to enable zero-shot cross-lingual NER. We adopt a multilingual labeled sequence translation model to project the tagged sequence back to the target language and label the target raw sentence.
  arXiv  Detail & Related papers  (2022-10-13T13:32:36Z)
• AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages [75.08199398141744]
  We present AmericasNLI, an extension of XNLI (Conneau et al.), to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%.
  arXiv  Detail & Related papers  (2021-04-18T05:32:28Z)
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders [111.65322283420805]
  We present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bi-directional EncodeR) AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model.
  arXiv  Detail & Related papers  (2020-10-15T18:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.