Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus

• URL: http://arxiv.org/abs/2605.00618v1
• Date: Fri, 01 May 2026 12:41:00 GMT
• Title: Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus
• Abstract summary: We investigate the extent to which cosine similarity between paragraph embeddings is invariant under machine translation.<n>We use the Corpus Manifesto of over 2,800 political party platforms in 28 languages translated to English via the EU eTranslation service.
• Score: 1.5364755220261122
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We investigate the extent to which cosine similarity between paragraph embeddings is invariant under machine translation, using the Manifesto Corpus of over 2,800 political party platforms in 28 languages translated to English via the EU eTranslation service. Rather than measuring translation-induced semantic shift directly we measure the stability of pairwise similarity relationships across embedding models, and use inter-model disagreement on original-language text as a calibrated invariance threshold. This yields a per-language non-inferiority test for four hypotheses about how translation interacts with embedding choice, with verdicts that distinguish languages where translation demonstrably preserves semantic structure from those where it demonstrably degrades it and from those where the available evidence does not resolve the question. The framework is corpus- and pipeline-agnostic and extends naturally to downstream tasks. Applied to our data, it identifies ten languages with translation invariance and four with detectable distortion.

Related papers

• Translation as a Decision Space: A Multi-Agent Perspective on Low-Resource Dialect Generation [3.2299961867400557]
  We conduct an empirical study on Turkish--Syrian Arabic translation using three agents.<n>We quantify structured behavioral displacement using dialect marker frequency, lexical proximity to standardized Arabic, and structural variance.<n>We argue that translation divergence across agents reveals latent decision flexibility within multilingual models.
  arXiv  Detail & Related papers  (2026-09-03T16:22:18Z)
• Language-Specific Gaps in AI Safety Training Datasets [0.0]
  Self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied.<n>We show that self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied.
  arXiv  Detail & Related papers  (2026-08-13T18:40:22Z)
• Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories [3.694051864521789]
  We recast fine-grained intertextuality extraction as an agentic task in which a large language model reads two text units in full.<n>We validate the approach on an exhaustive comparison of the Analects with the Book of Han, where three domain experts adjudicate a pooled multi-model candidate set into a benchmark of 2,533 intertextual pairs.
  arXiv  Detail & Related papers  (2026-07-30T02:34:23Z)
• XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics [64.77152900881724]
  We propose XQ-MEval, a semi-automatically built dataset covering nine translation directions.<n>We inject MQM-defined errors into gold translations automatically, filter them by native speakers for reliability, and merge errors to generate pseudo translations with controllable quality.<n>Using XQ-MEval, our experiments on nine representative metrics reveal the inconsistency between averaging and human judgment.
  arXiv  Detail & Related papers  (2026-04-16T12:27:10Z)
• Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages [0.22009842278462158]
  Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability.<n>We investigate evaluation reliability by holding generation conditions constant while varying target language.<n>Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages.
  arXiv  Detail & Related papers  (2026-02-02T16:27:32Z)
• Languages in Multilingual Speech Foundation Models Align Both Phonetically and Semantically [58.019484208091534]
  Cross-lingual alignment in pretrained language models (LMs) has enabled efficient transfer in text-based LMs.<n>It remains an open question whether findings and methods from text-based cross-lingual alignment apply to speech.
  arXiv  Detail & Related papers  (2025-05-26T07:21:20Z)
• PESTS: Persian_English Cross Lingual Corpus for Semantic Textual Similarity [5.439505575097552]
  Cross lingual semantic similarity models use a machine translation due to the unavailability of cross lingual semantic similarity dataset. For Persian, which is one of the low resource languages, the need for a model that can understand the context of two languages is felt more than ever. In this article, the corpus of semantic similarity between sentences in Persian and English languages has been produced for the first time by using linguistic experts.
  arXiv  Detail & Related papers  (2023-05-13T11:02:50Z)
• Interactive-Chain-Prompting: Ambiguity Resolution for Crosslingual Conditional Generation with Interaction [38.73550742775257]
  A source query in one language may yield several translation options in another language without any extra context. We propose a novel method interactive-chain prompting that reduces translations into a list of subproblems addressing ambiguities. We create a dataset exhibiting different linguistic phenomena which leads to ambiguities at inference for four languages.
  arXiv  Detail & Related papers  (2023-01-24T21:08:13Z)
• Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? [50.48082721476612]
  Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability. We investigate the distributions of grammatical relations induced from mBERT in the context of 24 typologically different languages.
  arXiv  Detail & Related papers  (2022-12-21T09:44:08Z)
• The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models [11.148662334602639]
  We analyze the position embeddings of existing language models and find strong evidence of translation invariance. We propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion.
  arXiv  Detail & Related papers  (2021-06-03T15:56:26Z)
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation [55.02832094101173]
  Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual similarity. This paper concerns ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations.
  arXiv  Detail & Related papers  (2020-05-03T22:10:23Z)
• Self-Attention with Cross-Lingual Position Representation [112.05807284056337]
  Position encoding (PE) is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. We augment SANs with emphcross-lingual position representations to model the bilingually aware latent structure for the input sentence.
  arXiv  Detail & Related papers  (2020-04-28T05:23:43Z)
• Translation Artifacts in Cross-lingual Transfer Learning [51.66536640084888]
  We show that machine translation can introduce subtle artifacts that have a notable impact in existing cross-lingual models. In natural language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them. We also improve the state-of-the-art in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points, respectively.
  arXiv  Detail & Related papers  (2020-04-09T17:54:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.