Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency

• URL: http://arxiv.org/abs/2603.17102v1
• Date: Tue, 17 Mar 2026 19:48:44 GMT
• Title: Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency
• Abstract summary: We propose leveraging cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs.<n>Our knowledge localization framework contrasts routing for sets of languages where the model correctly recalls information from languages where it fails.<n>We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases.
• Score: 25.655508607609615
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Modern LLMs continue to exhibit significant variance in behavior across languages, such as being able to recall factual information in some languages but not others. While typically studied as a problem to be mitigated, in this work, we propose leveraging this cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs. Our knowledge localization framework contrasts routing for sets of languages where the model correctly recalls information from languages where it fails. This allows us to isolate model components that play a functional role in answering about a piece of knowledge. Our method proceeds in two stages: (1) querying the model with difficult factual questions across a diverse set of languages to generate "success" and "failure" activation buckets and then (2) applying a statistical contrastive analysis to the MoE router logits to identify experts important for knowledge. To validate the necessity of this small number of experts for answering a knowledge question, we deactivate them and re-ask the question. We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases. Generally, this method provides a realistic and scalable knowledge localization approach to address increasingly complex LLMs.

Related papers

• Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts [25.655508607609615]
  We demonstrate that the perceived gap in knowledge transfer is primarily a script barrier.<n>We show that teaching two models to reason better reduces the cross-script transfer gap.
  arXiv  Detail & Related papers  (2026-03-17T18:57:26Z)
• CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention [9.76878200328024]
  Large Language Models (LLMs) often exhibit knowledge disparities across languages.<n>We introduce textitCausalAbstain, a method that helps LLMs determine whether to utilize multiple generated feedback responses.<n>Experiments demonstrate that textitCausalAbstain effectively selects helpful feedback and enhances abstention decisions with interpretability.
  arXiv  Detail & Related papers  (2025-05-31T11:35:31Z)
• How Do Multilingual Language Models Remember Facts? [50.13632788453612]
  We show that previously identified recall mechanisms in English largely apply to multilingual contexts.<n>We localize the role of language during recall, finding that subject enrichment is language-independent.<n>In decoder-only LLMs, FVs compose these two pieces of information in two separate stages.
  arXiv  Detail & Related papers  (2024-10-18T11:39:34Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora.<n>But can these models relate corresponding concepts across languages, i.e., be crosslingual?<n>This study evaluates state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements [59.71218039095155]
  Task of reading comprehension (RC) provides a primary means to assess language models' natural language understanding (NLU) capabilities.<n>If the context aligns with the models' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from internal information.<n>To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities.
  arXiv  Detail & Related papers  (2024-04-09T13:08:56Z)
• MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models [65.10456412127405]
  MLaKE is a benchmark for the adaptability of knowledge editing methods across five languages.<n>MLaKE aggregates fact chains from Wikipedia across languages and generates questions in both free-form and multiple-choice.<n>We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE.
  arXiv  Detail & Related papers  (2024-04-07T15:23:28Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.