IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following

• URL: http://arxiv.org/abs/2605.28218v1
• Date: Wed, 27 May 2026 09:35:27 GMT
• Title: IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following
• Abstract summary: Conventional metrics capture semantic fidelity but provide little signal on constraint adherence.<n>We introduce bench, a benchmark for multilingual translation instruction following seven languages.
• Score: 20.104094920745315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction following benchmarks ignore the cross-lingual nature of translation. We introduce \bench, a benchmark for multilingual translation instruction following covering seven languages, with 4,506 single-constraint and 2,838 multi-constraint items spanning six constraint dimensions and five compositional patterns with instructions issued in all seven languages. Constraints are split into a gating subset verified by deterministic checkers and a continuous subset scored by a rubric-based LLM judge, combined under a multiplicative rule that resists reward hacking. Evaluating 15 models reveals systematic gaps that prior protocols miss: Instruction following scales with size more sharply than translation quality, glossary and structured-format constraints dominate the difficulty gradient, and general instruction following rankings correlate only weakly with translation behavior. Our benchmark are available at https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench.

Related papers

• Generating Adversarial Texts for Machine Translation via GRPO [9.016860740260679]
  We propose a scalable reinforcement-learning-based approach for rewriting existing source texts.<n>On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability.<n> Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data.
  arXiv  Detail & Related papers  (2026-09-05T12:06:59Z)
• XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics [64.77152900881724]
  We propose XQ-MEval, a semi-automatically built dataset covering nine translation directions.<n>We inject MQM-defined errors into gold translations automatically, filter them by native speakers for reliability, and merge errors to generate pseudo translations with controllable quality.<n>Using XQ-MEval, our experiments on nine representative metrics reveal the inconsistency between averaging and human judgment.
  arXiv  Detail & Related papers  (2026-04-16T12:27:10Z)
• M3TQA: Massively Multilingual Multitask Table Question Answering [39.99483693397598]
  m3TQA-Instruct is a large-scale benchmark spanning 97 languages across diverse language families.<n>We construct m3TQA by curating 50 real-world tables in Chinese and English, then applying a robust six-step translation pipeline powered by DeepSeek and GPT-4o.<n>The benchmark includes 2,916 professionally annotated question-answering pairs across four tasks designed to evaluate nuanced table reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-22T09:57:40Z)
• XIFBench: Evaluating Large Language Models on Multilingual Instruction Following [59.549015333755186]
  Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications.<n>Existing evaluations lack fine-grained constraint analysis across diverse linguistic contexts.<n>We introduce XIFBench, a comprehensive benchmark for evaluating multilingual instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-03-10T17:07:52Z)
• MMTEB: Massive Multilingual Text Embedding Benchmark [85.18187649328792]
  We introduce the Massive Multilingual Text Embedding Benchmark (MMTEB)<n>MMTEB covers over 500 quality-controlled evaluation tasks across 250+ languages.<n>We develop several highly multilingual benchmarks, which we use to evaluate a representative set of models.
  arXiv  Detail & Related papers  (2025-02-19T10:13:43Z)
• Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following [51.18383180774354]
  We introduce Multi-IF, a new benchmark designed to assess Large Language Models' proficiency in following multi-turn and multilingual instructions. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models' multilingual capabilities.
  arXiv  Detail & Related papers  (2024-10-21T00:59:47Z)
• Decomposed Prompting for Machine Translation Between Related Languages using Large Language Models [55.35106713257871]
  We introduce DecoMT, a novel approach of few-shot prompting that decomposes the translation process into a sequence of word chunk translations. We show that DecoMT outperforms the strong few-shot prompting BLOOM model with an average improvement of 8 chrF++ scores across the examined languages.
  arXiv  Detail & Related papers  (2023-05-22T14:52:47Z)
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark [31.91964553419665]
  We present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. We demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.
  arXiv  Detail & Related papers  (2020-08-21T07:02:11Z)
• On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation [55.02832094101173]
  Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual similarity. This paper concerns ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations.
  arXiv  Detail & Related papers  (2020-05-03T22:10:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.