Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation

• URL: http://arxiv.org/abs/2604.19678v1
• Date: Tue, 21 Apr 2026 16:56:55 GMT
• Title: Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
• Abstract summary: Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning.<n>We study whether FVs exhibit language-agnosticity, using machine translation as a case study.
• Score: 18.767272825748456
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English$\rightarrow$Target direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. Ablation results show that removing the FV degrades translation across languages with limited impact on unrelated tasks. We further show that base-model FVs transfer to instruction-tuned variants and partially generalize from word-level to sentence-level translation.

Related papers

• Cross-lingual Functional Vectors for Emotion Detection in Large Language Models [2.3567984677073945]
  Function vectors (FVs) have emerged as a promising mechanism for steering the behavior of large language models.<n>We investigate the cross-lingual transferability of FVs using multilingual multi-label emotion recognition.
  arXiv  Detail & Related papers  (2026-08-30T07:08:36Z)
• Language Steering for Multilingual In-Context Learning [10.932074928744568]
  Large language models' performance on non-English languages remains substantially inferior to English.<n>We propose language vectors -- a training-free language steering approach.<n>We show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested.
  arXiv  Detail & Related papers  (2026-02-02T16:52:09Z)
• Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders [51.380449540006985]
  Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear.<n>Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language?<n>We analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs.
  arXiv  Detail & Related papers  (2025-11-13T22:51:06Z)
• Causal Language Control in Multilingual Transformers via Sparse Feature Steering [7.754609745940422]
  We investigate whether sparse autoencoder features can be leveraged to steer the generated language of multilingual language models.<n>We achieve controlled language shifts with up to 90% success, as measured by FastText language classification.<n>Our analysis reveals that language steering is most effective in mid-to-late transformer layers.
  arXiv  Detail & Related papers  (2025-07-17T06:49:16Z)
• Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation [16.368747052909214]
  Understanding representation transfer in multilingual neural machine translation (MNMT) can reveal the reason for the zero-shot translation deficiency.<n>We show that the encoder transfers the source language to the representational subspace of the target language instead of the language-agnostic state.<n>Based on our findings, we propose two methods: 1) low-rank language-specific embedding at the encoder, and 2) language-specific contrastive learning of the representation at the decoder.
  arXiv  Detail & Related papers  (2024-06-12T11:16:30Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)
• Improving Multilingual Translation by Representation and Gradient Regularization [82.42760103045083]
  We propose a joint approach to regularize NMT models at both representation-level and gradient-level. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance.
  arXiv  Detail & Related papers  (2021-09-10T10:52:21Z)
• First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT [2.2931318723689276]
  Cross-lingual transfer emerges from fine-tuning on a task of interest in one language and evaluating on a distinct language, not seen during the fine-tuning. We show that multilingual BERT can be viewed as the stacking of two sub-networks: a multilingual encoder followed by a task-specific language-agnostic predictor. While the encoder is crucial for cross-lingual transfer and remains mostly unchanged during fine-tuning, the task predictor has little importance on the transfer and can be red during fine-tuning.
  arXiv  Detail & Related papers  (2021-01-26T22:12:38Z)
• Improving Zero-Shot Translation by Disentangling Positional Information [24.02434897109097]
  We show that a main factor causing the language-specific representations is the positional correspondence to input tokens. We gain up to 18.5 BLEU points on zero-shot translation while retaining quality on supervised directions.
  arXiv  Detail & Related papers  (2020-12-30T12:20:41Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)
• Inducing Language-Agnostic Multilingual Representations [61.97381112847459]
  Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. We examine three approaches for this: (i) re-aligning the vector spaces of target languages to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering.
  arXiv  Detail & Related papers  (2020-08-20T17:58:56Z)
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers [62.637055980148816]
  Massively multilingual transformers pretrained with language modeling objectives have become a de facto default transfer paradigm for NLP. We show that cross-lingual transfer via massively multilingual transformers is substantially less effective in resource-lean scenarios and for distant languages.
  arXiv  Detail & Related papers  (2020-05-01T22:04:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.