MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning

• URL: http://arxiv.org/abs/2504.10160v1
• Date: Mon, 14 Apr 2025 12:14:18 GMT
• Title: MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning
• Abstract summary: We introduce MT-R1-Zero, the first open-source adaptation of the R1-Zero RL framework for machine translation (MT)<n>On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points.<n>Our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings.
• Score: 24.59797320737557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale reinforcement learning (RL) methods have proven highly effective in enhancing the reasoning abilities of large language models (LLMs), particularly for tasks with verifiable solutions such as mathematics and coding. However, applying this idea to machine translation (MT), where outputs are flexibly formatted and difficult to automatically evaluate with explicit rules, remains underexplored. In this work, we introduce MT-R1-Zero, the first open-source adaptation of the R1-Zero RL framework for MT without supervised fine-tuning or cold-start. We propose a rule-metric mixed reward mechanism to guide LLMs towards improved translation quality via emergent reasoning. On the WMT 24 English-Chinese benchmark, our MT-R1-Zero-3B-Mix achieves competitive performance, surpassing TowerInstruct-7B-v0.2 by an average of 1.26 points. Meanwhile, our MT-R1-Zero-7B-Mix attains a high average score of 62.25 across all metrics, placing it on par with advanced proprietary models such as GPT-4o and Claude-3.5-Sonnet, while the MT-R1-Zero-7B-Sem variant achieves state-of-the-art scores on semantic metrics. Moreover, our work exhibits strong generalization capabilities on out-of-distribution MT tasks, robustly supporting multilingual and low-resource settings. Extensive analysis of model behavior across different initializations and reward metrics offers pioneering insight into the critical role of reward design, LLM adaptability, training dynamics, and emergent reasoning patterns within the R1-Zero paradigm for MT. Our code is available at https://github.com/fzp0424/MT-R1-Zero.

Related papers

• North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+) [51.77806201331947]
  North Small Translate is an open-weight, LLM-based machine translation (MT) model with instruction-following capabilities.<n>North Small Translate is trained using difficulty sampling to obtain challenging documents.<n>The model achieves top MT performance across 50 languages in the class of models under 1T parameters.
  arXiv  Detail & Related papers  (2026-09-12T13:02:13Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization [74.04867639197445]
  MiroMind-M1 is a set of fully open-source RLMs built on the Qwen-2.5-based benchmarks.<n>Our models are trained in two stages: SFT on a carefully curated corpus of 719K math-reasoning problems with verified CoT trajectories, followed by RLVR on 62K challenging and verifiable problems.
  arXiv  Detail & Related papers  (2025-07-19T16:21:23Z)
• SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation [18.094515073693294]
  We propose a Simple Self-Rewarding (SSR) Reinforcement Learning (RL) framework for machine translation (MT) that is reference-free, fully online, and relies solely on self-judging rewards.<n>Training with SSR using 13K monolingual examples and Qwen-2.5-7B as the backbone, our model SSR-Zero-7B outperforms existing MT-specific LLMs.<n>By augmenting SSR with external supervision from COMET, our strongest model, SSR-X-Zero-7B, achieves state-of-the-art performance in English $leftrightarrow$ Chinese translation.
  arXiv  Detail & Related papers  (2025-05-22T13:08:25Z)
• ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning [77.41383117199227]
  We design a new reward modeling method that compares the translation results of the policy MT model with a strong LRM.<n>Using Qwen2.5-7B-Instruct as the backbone, the trained model achieves the new state-of-the-art performance in literary translation.<n>We extend our method to the multilingual settings with 11 languages.
  arXiv  Detail & Related papers  (2025-05-19T11:34:47Z)
• Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning [6.44608398856033]
  Rec-R1 bridges large language models (LLMs) with recommendation systems through closed-loop optimization. Unlike prompting and supervised fine-tuning (SFT), Rec-R1 directly optimize LLM generation using feedback from a fixed black-box recommendation model.
  arXiv  Detail & Related papers  (2025-03-31T16:36:00Z)
• MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling [7.980524378201173]
  Process reward models (PRMs) have shown success in complex reasoning tasks for large language models (LLMs) However, their application to machine translation (MT) remains underexplored due to the lack of systematic methodologies and evaluation benchmarks. We introduce textbfMT-RewardTree, a comprehensive framework for constructing, evaluating, and deploying process reward models in MT.
  arXiv  Detail & Related papers  (2025-03-15T13:04:51Z)
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models [24.45348222168512]
  We propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability.<n>Our model achieves an average improvement of $sim$6% across various multimodal math reasoning benchmarks.<n>Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1.
  arXiv  Detail & Related papers  (2025-03-09T20:06:45Z)
• R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learning [23.721573333602677]
  This paper introduces R1-Translator (R1-T1), a novel framework to achieve inference-time reasoning for general machine translation (MT) via reinforcement learning (RL)<n>Our approach pioneers three innovations: (1) extending reasoning-based translation beyond MT sub-tasks to six languages and diverse tasks (e.g., legal/medical domain adaptation, idiom resolution); and (2) formalizing six expert-curated CoT templates that mirror hybrid human strategies like context-aware paraphrasing and back translation.<n> Experimental results indicate a steady translation performance improvement in 11 languages and 40 translation directions on Flores-101 test set, especially on the
  arXiv  Detail & Related papers  (2025-02-27T03:57:00Z)
• Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling [52.34735382627312]
  Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. Existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. We present T1 to scale reinforcement learning by encouraging exploration and understand inference scaling.
  arXiv  Detail & Related papers  (2025-01-20T18:33:33Z)
• Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024 [61.189875635090225]
  Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST)
  arXiv  Detail & Related papers  (2024-06-24T16:38:17Z)
• Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level [9.699022347910121]
  General-purpose Large Language Models (LLMs) have achieved remarkable advancements in machine translation (MT) by leveraging extensive web content. However, translation-specific LLMs are built by pre-training on domain-specific monolingual corpora and fine-tuning with human-annotated translation data. We develop MT-Ladder, a novel model-agnostic and cost-effective tool to refine the performance of general LLMs for MT.
  arXiv  Detail & Related papers  (2024-06-22T05:33:35Z)
• Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation [50.00235162432848]
  We train ALMA models with only 22K parallel sentences and 12M parameters. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4.
  arXiv  Detail & Related papers  (2024-01-16T15:04:51Z)
• On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation [80.16548523140025]
  Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT) We propose to combine their complementarities with a model fusion algorithm that utilizes optimal transport to align neurons between PT and RI. Experiments on two resource-rich translation benchmarks, WMT'17 English-Chinese (20M) and WMT'19 English-German (36M), show that PT and RI could be nicely complementary to each other.
  arXiv  Detail & Related papers  (2022-09-07T17:23:08Z)
• Improving Rare Word Recognition with LM-aware MWER Training [50.241159623691885]
  We introduce LMs in the learning of hybrid autoregressive transducer (HAT) models in the discriminative training framework. For the shallow fusion setup, we use LMs during both hypotheses generation and loss computation, and the LM-aware MWER-trained model achieves 10% relative improvement. For the rescoring setup, we learn a small neural module to generate per-token fusion weights in a data-dependent manner.
  arXiv  Detail & Related papers  (2022-04-15T17:19:41Z)
• The USYD-JD Speech Translation System for IWSLT 2021 [85.64797317290349]
  This paper describes the University of Sydney& JD's joint submission of the IWSLT 2021 low resource speech translation task. We trained our models with the officially provided ASR and MT datasets. To achieve better translation performance, we explored the most recent effective strategies, including back translation, knowledge distillation, multi-feature reranking and transductive finetuning.
  arXiv  Detail & Related papers  (2021-07-24T09:53:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.