Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?

• URL: http://arxiv.org/abs/2408.10811v1
• Date: Tue, 20 Aug 2024 13:05:41 GMT
• Title: Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?
• Authors: Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, Sadao Kurohashi,
• Abstract summary: We investigate whether non-English-centric LLMs, despite their strong performance, think' in their respective dominant language. We term such languages as internal $textbflatent languages$.
• Score: 40.53443067505763
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this study, we investigate whether non-English-centric LLMs, despite their strong performance, `think' in their respective dominant language: more precisely, `think' refers to how the representations of intermediate layers, when un-embedded into the vocabulary space, exhibit higher probabilities for certain dominant languages during generation. We term such languages as internal $\textbf{latent languages}$. We examine the latent language of three typical categories of models for Japanese processing: Llama2, an English-centric model; Swallow, an English-centric model with continued pre-training in Japanese; and LLM-jp, a model pre-trained on balanced English and Japanese corpora. Our empirical findings reveal that, unlike Llama2 which relies exclusively on English as the internal latent language, Japanese-specific Swallow and LLM-jp employ both Japanese and English, exhibiting dual internal latent languages. For any given target language, the model preferentially activates the latent language most closely related to it. In addition, we explore how intermediate layers respond to questions involving cultural conflicts between latent internal and target output languages. We further explore how the language identity shifts across layers while keeping consistent semantic meaning reflected in the intermediate layer representations. This study deepens the understanding of non-English-centric large language models, highlighting the intricate dynamics of language representation within their intermediate layers.

Related papers

• How does a Multilingual LM Handle Multiple Languages? [0.0]
  This study critically examines capabilities in multilingual understanding, semantic representation, and cross-lingual knowledge transfer. It assesses semantic similarity by analyzing multilingual word embeddings for consistency using cosine similarity. It examines BLOOM-1.7B and Qwen2 through Named Entity Recognition and sentence similarity tasks to understand their linguistic structures.
  arXiv  Detail & Related papers  (2025-02-06T18:08:14Z)
• Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages [15.203789021094982]
  In large language models (LLMs), how are multiple languages learned and encoded? We train sparse autoencoders on Llama-3-8B and Aya-23-8B, and demonstrate that abstract grammatical concepts are often encoded in feature directions shared across many languages.
  arXiv  Detail & Related papers  (2025-01-10T21:18:21Z)
• Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense [30.62699081329474]
  We introduce a novel benchmark for cross-lingual sense disambiguation, StingrayBench. We collect false friends in four language pairs, namely Indonesian-Malay, Indonesian-Tagalog, Chinese-Japanese, and English-German. In our analysis of various models, we observe they tend to be biased toward higher-resource languages.
  arXiv  Detail & Related papers  (2024-10-28T22:09:43Z)
• How Do Multilingual Language Models Remember Facts? [50.13632788453612]
  We show that previously identified recall mechanisms in English largely apply to multilingual contexts. We localize the role of language during recall, finding that subject enrichment is language-independent. In decoder-only LLMs, FVs compose these two pieces of information in two separate stages.
  arXiv  Detail & Related papers  (2024-10-18T11:39:34Z)
• Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models [11.423589362950812]
  Large language models (LLMs) have demonstrated remarkable performance, particularly in multilingual contexts. Recent studies suggest that LLMs can transfer skills learned in one language to others, but the internal mechanisms behind this ability remain unclear. This paper provides insights into the internal workings of LLMs, offering a foundation for future improvements in their cross-lingual capabilities.
  arXiv  Detail & Related papers  (2024-10-15T15:49:15Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  Lens is a novel approach to enhance multilingual capabilities of large language models (LLMs) It operates by manipulating the hidden representations within the language-agnostic and language-specific subspaces from top layers of LLMs. It achieves superior results with much fewer computational resources compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Decomposed Prompting: Unveiling Multilingual Linguistic Structure Knowledge in English-Centric Large Language Models [12.700783525558721]
  English-centric Large Language Models (LLMs) like GPT-3 and LLaMA display a remarkable ability to perform multilingual tasks. This paper introduces the decomposed prompting approach to probe the linguistic structure understanding of these LLMs in sequence labeling tasks.
  arXiv  Detail & Related papers  (2024-02-28T15:15:39Z)
• Multi-lingual and Multi-cultural Figurative Language Understanding [69.47641938200817]
  Figurative language permeates human communication, but is relatively understudied in NLP. We create a dataset for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data.
  arXiv  Detail & Related papers  (2023-05-25T15:30:31Z)
• Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure [54.01613740115601]
  We study three language properties: constituent order, composition and word co-occurrence. Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.
  arXiv  Detail & Related papers  (2022-03-16T07:09:35Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.