Language of Thought Shapes Output Diversity in Large Language Models

• URL: http://arxiv.org/abs/2601.11227v1
• Date: Fri, 16 Jan 2026 12:14:16 GMT
• Title: Language of Thought Shapes Output Diversity in Large Language Models
• Authors: Shaoyang Xu, Wenxuan Zhang,
• Abstract summary: We show that different thinking languages occupy distinct regions in a model's thinking space.<n>We demonstrate that switching the thinking language from English to non-English languages consistently increases output diversity.<n>We show that these findings translate into practical benefits in pluralistic alignment scenarios.
• Score: 19.140956219362874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Output diversity is crucial for Large Language Models as it underpins pluralism and creativity. In this work, we reveal that controlling the language used during model thinking-the language of thought-provides a novel and structural source of output diversity. Our preliminary study shows that different thinking languages occupy distinct regions in a model's thinking space. Based on this observation, we study two repeated sampling strategies under multilingual thinking-Single-Language Sampling and Mixed-Language Sampling-and conduct diversity evaluation on outputs that are controlled to be in English, regardless of the thinking language used. Across extensive experiments, we demonstrate that switching the thinking language from English to non-English languages consistently increases output diversity, with a clear and consistent positive correlation such that languages farther from English in the thinking space yield larger gains. We further show that aggregating samples across multiple thinking languages yields additional improvements through compositional effects, and that scaling sampling with linguistic heterogeneity expands the model's diversity ceiling. Finally, we show that these findings translate into practical benefits in pluralistic alignment scenarios, leading to broader coverage of cultural knowledge and value orientations in LLM outputs. Our code is publicly available at https://github.com/iNLP-Lab/Multilingual-LoT-Diversity.

Related papers

• The Emergence of Abstract Thought in Large Language Models Beyond Any Language [95.50197866832772]
  Large language models (LLMs) function effectively across a diverse range of languages.<n>Preliminary studies observe that the hidden activations of LLMs often resemble English, even when responding to non-English prompts.<n>Recent results show strong multilingual performance, even surpassing English performance on specific tasks in other languages.
  arXiv  Detail & Related papers  (2025-06-11T16:00:54Z)
• When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners [111.50503126693444]
  We show that language-specific ablation consistently boosts multilingual reasoning performance.<n>Compared to post-training, our training-free ablation achieves comparable or superior results with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-05-21T08:35:05Z)
• CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMs [1.2057938662974816]
  Large Language Models (LLMs) develop cross-lingual abilities despite being trained on limited parallel data.<n>We introduce CoCo-CoLa, a novel metric to evaluate language adherence in multilingual LLMs.
  arXiv  Detail & Related papers  (2025-02-18T03:03:53Z)
• A Survey of Multilingual Reasoning in Language Models [30.140967158580892]
  This survey provides the first in-depth review of multilingual reasoning in language models.<n>We provide an overview of the standard data resources used for training multilingual reasoning in LMs.<n>We analyze various state-of-the-art methods and their performance on these benchmarks.
  arXiv  Detail & Related papers  (2025-02-13T16:25:16Z)
• Benchmarking Linguistic Diversity of Large Language Models [14.824871604671467]
  This paper emphasizes the importance of examining the preservation of human linguistic richness by language models.<n>We propose a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives.
  arXiv  Detail & Related papers  (2024-12-13T16:46:03Z)
• LOLA -- An Open-Source Massively Multilingual Large Language Model [1.5704590739448838]
  LOLA is a massively multilingual large language model trained on more than 160 languages.<n>Our architectural and implementation choices address the challenge of harnessing linguistic diversity.<n>We show how the learned expert-routing mechanism exploits implicit phylogenetic patterns to potentially alleviate the curse of multilinguality.
  arXiv  Detail & Related papers  (2024-09-17T15:23:08Z)
• The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments [57.273662221547056]
  In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. We observe that the existence of a predominant language during training boosts the performance of less frequent languages. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
  arXiv  Detail & Related papers  (2024-04-11T17:58:05Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer [101.58431011820755]
  We study gender bias in multilingual embeddings and how it affects transfer learning for NLP applications. We create a multilingual dataset for bias analysis and propose several ways for quantifying bias in multilingual representations.
  arXiv  Detail & Related papers  (2020-05-02T04:34:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.