Multilingual Embedding Probes Fail to Generalize Across Learner Corpora

• URL: http://arxiv.org/abs/2604.07095v1
• Date: Wed, 08 Apr 2026 13:47:54 GMT
• Title: Multilingual Embedding Probes Fail to Generalize Across Learner Corpora
• Abstract summary: We train probes on hidden-state activations to predict CEFR proficiency levels from learner texts across nine corpora and seven languages.<n>Under in-distribution evaluation, probes achieve strong performance, substantially outperforming the surface baseline.<n>In cross-corpus evaluation performance collapses across all probe types and model sizes.<n>These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency.
• Score: 0.17188280334580197
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Do multilingual embedding models encode a language-general representation of proficiency? We investigate this by training linear and non-linear probes on hidden-state activations from Qwen3-Embedding (0.6B, 4B, 8B) to predict CEFR proficiency levels from learner texts across nine corpora and seven languages. We compare five probing architectures against a baseline trained on surface-level text features. Under in-distribution evaluation, probes achieve strong performance ($QWK\approx0.7$), substantially outperforming the surface baseline, with middle layers consistently yielding the best predictions. However, in cross-corpus evaluation performance collapses across all probe types and model sizes. Residual analysis reveals that out-of-distribution probes converge towards predicting uniformly distributed labels, indicating that the learned mappings capture corpus-specific distributional properties (topic, language, task type, rating methodology) rather than an abstract, transferable proficiency dimension. These results suggest that current multilingual embeddings do not straightforwardly encode language-general proficiency, with implications for representation-based approaches to proficiency-adaptive language technology.

Related papers

• What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models [0.19116784879310025]
  Cross-lingual information retrieval is challenging due to disparities in resources, scripts, and weak cross-lingual semantic alignment in embedding models.<n>Existing pipelines often rely on translation and monolingual retrievals, which add computational overhead and noise, performance.<n>This work systematically evaluates four intervention types, namely document translation, multilingual dense retrieval with pretrained encoders, contrastive learning at word, phrase, and query-document levels, and cross-encoder re-ranking, across three benchmark datasets.
  arXiv  Detail & Related papers  (2025-11-24T17:17:40Z)
• Low-Resource English-Tigrinya MT: Leveraging Multilingual Models, Custom Tokenizers, and Clean Evaluation Benchmarks [6.177998679139308]
  Despite advances in Neural Machine Translation (NMT), low-resource languages like Tigrinya remain underserved.<n>This paper investigates transfer learning techniques using multilingual pretrained models to enhance translation quality for morphologically rich, low-resource languages.
  arXiv  Detail & Related papers  (2025-09-24T15:02:57Z)
• Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing [68.47787275021567]
  Cross-lingual semantic parsing transfers parsing capability from a high-resource language (e.g., English) to low-resource languages with scarce training data. We propose a new approach to cross-lingual semantic parsing by explicitly minimizing cross-lingual divergence between latent variables using Optimal Transport.
  arXiv  Detail & Related papers  (2023-07-09T04:52:31Z)
• T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classification [50.675552118811]
  Cross-lingual text classification is typically built on large-scale, multilingual language models (LMs) pretrained on a variety of languages of interest. We propose revisiting the classic "translate-and-test" pipeline to neatly separate the translation and classification stages.
  arXiv  Detail & Related papers  (2023-06-08T07:33:22Z)
• Enhancing Pashto Text Classification using Language Processing Techniques for Single And Multi-Label Analysis [0.0]
  This study aims to establish an automated classification system for Pashto text. The study achieved an average testing accuracy rate of 94%. The use of pre-trained language representation models, such as DistilBERT, showed promising results.
  arXiv  Detail & Related papers  (2023-05-04T23:11:31Z)
• Retrieval-based Disentangled Representation Learning with Natural Language Supervision [61.75109410513864]
  We present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employ a bi-encoder model to represent both data and natural language in a vocabulary space, enabling the model to distinguish intrinsic dimensions that capture characteristics within data through its natural language counterpart, thus disentanglement.
  arXiv  Detail & Related papers  (2022-12-15T10:20:42Z)
• Systematic Investigation of Strategies Tailored for Low-Resource Settings for Sanskrit Dependency Parsing [14.416855042499945]
  Existing state of the art approaches for Sanskrit Dependency Parsing (SDP) are hybrid in nature. purely data-driven approaches do not match the performance of hybrid approaches due to labelled data sparsity. We experiment with five strategies, namely, data augmentation, sequential transfer learning, cross-lingual/mono-lingual pretraining, multi-task learning and self-training. Our proposed ensembled system outperforms the purely data-driven state of the art system by 2.8/3.9 points (Unlabelled Attachment Score (UAS)/Labelled Attachment Score (LAS)) absolute gain
  arXiv  Detail & Related papers  (2022-01-27T08:24:53Z)
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension [51.953428342923885]
  We develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning mechanism is developed to learn the fine difference between the accurate answer and other candidates.
  arXiv  Detail & Related papers  (2021-12-09T07:31:15Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• Cross-lingual Text Classification with Heterogeneous Graph Neural Network [2.6936806968297913]
  Cross-lingual text classification aims at training a classifier on the source language and transferring the knowledge to target languages. Recent multilingual pretrained language models (mPLM) achieve impressive results in cross-lingual classification tasks. We propose a simple yet effective method to incorporate heterogeneous information within and across languages for cross-lingual text classification.
  arXiv  Detail & Related papers  (2021-05-24T12:45:42Z)
• TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing [73.16475763422446]
  We propose a multilingual robustness evaluation platform for NLP tasks (TextFlint) It incorporates universal text transformation, task-specific transformation, adversarial attack, subpopulation, and their combinations to provide comprehensive robustness analysis. TextFlint generates complete analytical reports as well as targeted augmented data to address the shortcomings of the model's robustness.
  arXiv  Detail & Related papers  (2021-03-21T17:20:38Z)
• A Bayesian Multilingual Document Model for Zero-shot Topic Identification and Discovery [1.9215779751499527]
  The model is an extension of BaySMM [Kesiraju et al 2020] to the multilingual scenario. We propagate the learned uncertainties through linear classifiers that benefit zero-shot cross-lingual topic identification. We revisit cross-lingual topic identification in zero-shot settings by taking a deeper dive into current datasets.
  arXiv  Detail & Related papers  (2020-07-02T19:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.