LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

• URL: http://arxiv.org/abs/2604.18490v1
• Date: Mon, 20 Apr 2026 16:41:37 GMT
• Title: LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
• Abstract summary: We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT.<n>LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels.<n>We produce 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores.
• Score: 27.415791404495696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely language-agnostic. However, they often fail to capture dialect- and culture-specific errors in diglossic languages (e.g., Arabic), where translation failures stem from mismatches in language variety, content coverage, and pragmatic appropriateness rather than surface form alone.We introduce LQM: Linguistically Motivated Multidimensional Quality Metrics for MT. LQM is a hierarchical error taxonomy for diagnosing MT errors through six linguistically grounded levels: sociolinguistics, pragmatics, semantics, morphosyntax, orthography, and graphetics (Figure 1). We construct a bidirectional parallel corpus of 3,850 sentences (550 per variety) spanning seven Arabic dialects (Egyptian, Emirati, Jordanian, Mauritanian, Moroccan, Palestinian, and Yemeni), derived from conversational, culturally rich content. We evaluate six LLMs in a zero-shot setting and conduct expert span-level human annotation using LQM, producing 6,113 labeled error spans across 3,495 unique erroneous sentences, along with severity-weighted quality scores. We complement this analysis with an automatic metric (spBLEU). Though validated here on Arabic, LQM is a language-agnostic framework designed to be easily applied to or adapted for other languages. LQM annotated errors data, prompts, and annotation guidelines are publicly available at https://github.com/UBC-NLP/LQM_MT.

Related papers

• M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models [0.0]
  M-GATE is a benchmark of linguistic proficiency spanning 30 typologically diverse languages.<n>We evaluate over 50 models in more than 80 configurations.
  arXiv  Detail & Related papers  (2026-08-04T15:16:05Z)
• The Role of Prompt Language and Translation-Theory-Driven Prompts in Large Language Models: A Case Study on Spanish-Chinese Journalistic Translation [13.47712128273878]
  This study examines how prompt language and translation theory-driven prompt design influence the quality of Spanish-Chinese journalistic translations.<n>A parallel corpus of four editorials from El Pais was translated under 48 experimental conditions.
  arXiv  Detail & Related papers  (2026-07-03T09:59:36Z)
• IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following [20.104094920745315]
  Conventional metrics capture semantic fidelity but provide little signal on constraint adherence.<n>We introduce bench, a benchmark for multilingual translation instruction following seven languages.
  arXiv  Detail & Related papers  (2026-05-27T09:35:27Z)
• XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics [64.77152900881724]
  We propose XQ-MEval, a semi-automatically built dataset covering nine translation directions.<n>We inject MQM-defined errors into gold translations automatically, filter them by native speakers for reliability, and merge errors to generate pseudo translations with controllable quality.<n>Using XQ-MEval, our experiments on nine representative metrics reveal the inconsistency between averaging and human judgment.
  arXiv  Detail & Related papers  (2026-04-16T12:27:10Z)
• Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language [1.0742675209112622]
  We examine whether structured prompts alone can elicit basic conversational ability under controlled prompting.<n>We combine explicit grammar documentation, negative constraints to suppress high-probability tokens from related languages, romanization standardization, and quality-controlled synthetic data generation via self-play.<n>Our approach reduces vocabulary contamination from 80% to 5% while achieving 85% grammatical accuracy.
  arXiv  Detail & Related papers  (2026-02-17T06:20:09Z)
• SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages [4.298924990628206]
  SiniticMTError is a novel dataset that builds on existing parallel corpora to provide error span, error type, and error severity annotations.<n>Our dataset serves as a resource for the MT community to utilize in fine-tuning models with error detection capabilities.<n>We report our rigorous annotation process by native speakers, with analyses on inter-annotator agreement, iterative feedback, and patterns in error type and severity.
  arXiv  Detail & Related papers  (2025-09-24T20:50:09Z)
• HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark [54.73504952691398]
  We set out to deliver a Hebrew Machine Reading dataset as extractive Questioning.<n>The morphologically rich nature of Hebrew poses a challenge to this endeavor.<n>We devise a novel set of guidelines, a controlled crowdsourcing protocol, and revised evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-03T15:53:01Z)
• Grammatical Error Correction for Low-Resource Languages: The Case of Zarma [8.40484790921164]
  Grammatical error correction aims to improve quality and readability of texts.<n>We present a study on GEC for Zarma, spoken by over five million in West Africa.<n>We compare three approaches: rule-based methods, machine translation (MT) models, and large language models.
  arXiv  Detail & Related papers  (2024-10-20T23:51:36Z)
• MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators [53.91199933655421]
  Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment.<n>We introduce a universal and training-free framework, $textbfMQM-APE, based on the idea of filtering out non-impactful errors.<n>Experiments show that our approach consistently improves both the reliability and quality of error spans against GEMBA-MQM.
  arXiv  Detail & Related papers  (2024-09-22T06:43:40Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• CLSE: Corpus of Linguistically Significant Entities [58.29901964387952]
  We release a Corpus of Linguistically Significant Entities (CLSE) annotated by experts. CLSE covers 74 different semantic types to support various applications from airline ticketing to video games. We create a linguistically representative NLG evaluation benchmark in three languages: French, Marathi, and Russian.
  arXiv  Detail & Related papers  (2022-11-04T12:56:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.