Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

• URL: http://arxiv.org/abs/2602.22453v2
• Date: Fri, 27 Feb 2026 17:36:39 GMT
• Title: Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
• Authors: Shaswat Patel, Vishvesh Trivedi, Yue Han, Yihuai Hong, Eunsol Choi,
• Abstract summary: Retrieval heads are often shared across multiple languages.<n>Retrieval-Transition heads govern the transition to specific target-language output.<n>Our work advances understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.
• Score: 33.242977481016375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has identified a subset of attention heads in Transformer as retrieval heads, which are responsible for retrieving information from the context. In this work, we first investigate retrieval heads in multilingual contexts. In multilingual language models, we find that retrieval heads are often shared across multiple languages. Expanding the study to cross-lingual setting, we identify Retrieval-Transition heads(RTH), which govern the transition to specific target-language output. Our experiments reveal that RTHs are distinct from retrieval heads and more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuaD) and two model families (Qwen-2.5 and Llama-3.1), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Our work advances understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.

Related papers

• Bridging Language Gaps: Advances in Cross-Lingual Information Retrieval with Multilingual LLMs [0.19116784879310025]
  Cross-lingual information retrieval (CLIR) addresses the challenge of retrieving relevant documents written in languages different from that of the original query.<n>Recent advances have shifted from translation-based methods toward embedding-based approaches.<n>This survey provides a comprehensive overview of developments from early translation-based methods to state-of-the-art embedding-driven and generative techniques.
  arXiv  Detail & Related papers  (2025-10-01T13:50:05Z)
• XRAG: Cross-lingual Retrieval-Augmented Generation [21.548347969135254]
  XRAG is designed to evaluate the generation abilities of LLMs in cross-lingual Retrieval-Augmented Generation settings.<n>XRAG is constructed from recent news articles to ensure that its questions require external knowledge to be answered.
  arXiv  Detail & Related papers  (2025-05-15T08:47:55Z)
• Improving Retrieval-Augmented Neural Machine Translation with Monolingual Data [18.150384435635477]
  In many settings, monolingual corpora in the target language are often available.<n>We design improved cross-lingual retrieval systems, trained with both sentence level and word-level matching objectives.<n>We also showcase our method on a real-world settings, using much larger monolingual and observe strong improvements over both the baseline setting and general-purpose cross-lingual retrievers.
  arXiv  Detail & Related papers  (2025-04-30T15:41:03Z)
• Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Task [89.45111250272559]
  Retrieval-augmented generation (RAG) has become a cornerstone of contemporary NLP.<n>This paper investigates the effectiveness of RAG across multiple languages by proposing novel approaches for multilingual open-domain question-answering.
  arXiv  Detail & Related papers  (2025-04-04T17:35:43Z)
• mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval [61.17793165194077]
  We introduce mFollowIR, a benchmark for measuring instruction-following ability in retrieval models.<n>We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance.<n>We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting.
  arXiv  Detail & Related papers  (2025-01-31T16:24:46Z)
• How Do Multilingual Language Models Remember Facts? [50.13632788453612]
  We show that previously identified recall mechanisms in English largely apply to multilingual contexts.<n>We localize the role of language during recall, finding that subject enrichment is language-independent.<n>In decoder-only LLMs, FVs compose these two pieces of information in two separate stages.
  arXiv  Detail & Related papers  (2024-10-18T11:39:34Z)
• Cross-Lingual Transfer Robustness to Lower-Resource Languages on Adversarial Datasets [4.653113033432781]
  Cross-lingual transfer capabilities of Multilingual Language Models (MLLMs) are investigated. Our research provides valuable insights into cross-lingual transfer and its implications for NLP applications.
  arXiv  Detail & Related papers  (2024-03-29T08:47:15Z)
• MELA: Multilingual Evaluation of Linguistic Acceptability [7.524375463656369]
  We present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability -- MELA, with 46K samples covering 10 languages. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial.
  arXiv  Detail & Related papers  (2023-11-15T15:25:28Z)
• Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog [67.20796950016735]
  Multi2WOZ dataset spans four typologically diverse languages: Chinese, German, Arabic, and Russian. We introduce a new framework for multilingual conversational specialization of pretrained language models (PrLMs) that aims to facilitate cross-lingual transfer for arbitrary downstream TOD tasks. Our experiments show that, in most setups, the best performance entails the combination of (I) conversational specialization in the target language and (ii) few-shot transfer for the concrete TOD task.
  arXiv  Detail & Related papers  (2022-05-20T18:35:38Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)
• Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks [9.913751245347429]
  We show that pruning a number of attention heads in a multi-lingual Transformer-based model has, in general, positive effects on its performance in cross-lingual and multi-lingual tasks. For comprehensiveness, we examine two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and XLM-R, on three tasks across 9 languages each.
  arXiv  Detail & Related papers  (2021-08-18T20:17:46Z)
• Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval [51.60862829942932]
  We present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. For sentence-level CLIR, we demonstrate that state-of-the-art performance can be achieved. However, the peak performance is not met using the general-purpose multilingual text encoders off-the-shelf', but rather relying on their variants that have been further specialized for sentence understanding tasks.
  arXiv  Detail & Related papers  (2021-01-21T00:15:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.