Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders

• URL: http://arxiv.org/abs/2603.19771v1
• Date: Fri, 20 Mar 2026 09:02:38 GMT
• Title: Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders
• Abstract summary: We construct a unified trilingual corpus of parallel English, Hindi, and Romanized code-mixed sentences.<n>We find that while standard models align English and Hindi well, code-mixed inputs remain loosely connected to either language.
• Score: 0.9523331544264052
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Multilingual encoder-based language models are widely adopted for code-mixed analysis tasks, yet we know surprisingly little about how they represent code-mixed inputs internally - or whether those representations meaningfully connect to the constituent languages being mixed. Using Hindi-English as a case study, we construct a unified trilingual corpus of parallel English, Hindi (Devanagari), and Romanized code-mixed sentences, and probe cross-lingual representation alignment across standard multilingual encoders and their code-mixed adapted variants via CKA, token-level saliency, and entropy-based uncertainty analysis. We find that while standard models align English and Hindi well, code-mixed inputs remain loosely connected to either language - and that continued pre-training on code-mixed data improves English-code-mixed alignment at the cost of English-Hindi alignment. Interpretability analyses further reveal a clear asymmetry: models process code-mixed text through an English-dominant semantic subspace, while native-script Hindi provides complementary signals that reduce representational uncertainty. Motivated by these findings, we introduce a trilingual post-training alignment objective that brings code-mixed representations closer to both constituent languages simultaneously, yielding more balanced cross-lingual alignment and downstream gains on sentiment analysis and hate speech detection - showing that grounding code-mixed representations in their constituent languages meaningfully helps cross-lingual understanding.

Related papers

• IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing [3.4293055914377564]
  Traditional language identification models, designed for monolingual text, are not well suited for token-level language identification in code-mixed settings.<n>We formulate the task as a sequence labeling problem and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa.<n>We propose two approaches of code-mixed generation using parallel sentences of three languages.
  arXiv  Detail & Related papers  (2026-09-10T17:36:12Z)
• Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR [11.507193494503754]
  We examine how knowledge recall consistency varies across the spectrum of English, code-mixed, and native Indian language inputs.<n>We find that the native-language accuracy gap to English can reach $sim$0.50, while code-mixed inputs close most of it.<n>Motivated by this, we evaluate several prompting strategies that vary in how language conversion is exposed.
  arXiv  Detail & Related papers  (2026-05-28T09:06:59Z)
• Evaluating Multilingual and Code-Switched Alignment in LLMs via Synthetic Natural Language Inference [2.172419551358714]
  Large language models (LLMs) are increasingly applied in multilingual contexts, yet their capacity for consistent, logically grounded alignment across languages remains underexplored.<n>We present a framework for multilingual natural language inference that generates synthetic, logic-based premise-hypothesis pairs and translates them into a typologically diverse set of languages.<n>Code-switching does not degrade, and can even improve, performance, suggesting that translation-induced lexical variation may serve as a regularization signal.
  arXiv  Detail & Related papers  (2025-08-20T14:30:34Z)
• Disentangling Codemixing in Chats: The NUS ABC Codemixed Corpus [11.518751071307745]
  Code-mixing involves the seamless integration of linguistic elements from multiple languages within a single discourse.<n>There has been a lack of publicly available corpora that are author-labeled and suitable for modeling human conversations and relationships.<n>This study introduces the first labeled and general-purpose corpus for understanding code-mixing in context while maintaining rigorous privacy and ethical standards.
  arXiv  Detail & Related papers  (2025-05-31T01:09:04Z)
• Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training [58.696660064190475]
  We find that the existence of code-switching, alternating between different languages within a context, is key to multilingual capabilities.<n>To better explore the power of code-switching for language alignment during pre-training, we investigate the strategy of synthetic code-switching.
  arXiv  Detail & Related papers  (2025-04-02T15:09:58Z)
• Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment [50.80949663719335]
  multilingual sentence encoders (MSEs) are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space.<n>MSEs are subject to curse of multilinguality, a loss of monolingual representational accuracy due to parameter sharing.<n>We train the cross-lingual adapters with two different types of data to resolve the conflicting requirements of different cross-lingual tasks.
  arXiv  Detail & Related papers  (2024-07-20T13:56:39Z)
• Understanding Cross-Lingual Alignment -- A Survey [52.572071017877704]
  Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field.
  arXiv  Detail & Related papers  (2024-04-09T11:39:53Z)
• The Effect of Alignment Objectives on Code-Switching Translation [0.0]
  We are proposing a way of training a single machine translation model that is able to translate monolingual sentences from one language to another. This model can be considered a bilingual model in the human sense.
  arXiv  Detail & Related papers  (2023-09-10T14:46:31Z)
• Leveraging Language Identification to Enhance Code-Mixed Text Classification [0.7340017786387767]
  Existing deep-learning models do not take advantage of the implicit language information in code-mixed text. Our study aims to improve BERT-based models performance on low-resource Code-Mixed Hindi-English datasets.
  arXiv  Detail & Related papers  (2023-06-08T06:43:10Z)
• Zero-Shot Cross-lingual Semantic Parsing [56.95036511882921]
  We study cross-lingual semantic parsing as a zero-shot problem without parallel data for 7 test languages. We propose a multi-task encoder-decoder model to transfer parsing knowledge to additional languages using only English-Logical form paired data. Our system frames zero-shot parsing as a latent-space alignment problem and finds that pre-trained models can be improved to generate logical forms with minimal cross-lingual transfer penalty.
  arXiv  Detail & Related papers  (2021-04-15T16:08:43Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.